Nov 02 01:24:11.141355 ip-10-0-134-183 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:24:11.489538 ip-10-0-134-183 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:11.489538 ip-10-0-134-183 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:24:11.489538 ip-10-0-134-183 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:11.489538 ip-10-0-134-183 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:11.493212 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493135    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:11.493212 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493212    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:11.493212 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493216    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493219    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493222    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493225    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493228    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493231    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493234    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493236    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493239    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493242    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493244    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493248    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493251    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493254    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493258    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493262    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493265    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493267    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493270    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:11.493304 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493272    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493275    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493277    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493280    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493282    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493285    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493294    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493297    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493300    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493303    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493305    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493308    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493311    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493314    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493317    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493319    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493324    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493326    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493330    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493332    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:11.493771 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493335    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493338    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493340    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493344    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493348    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493351    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493353    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493356    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493359    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493361    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493364    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493366    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493368    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493371    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493374    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493377    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493380    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493382    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493385    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493387    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:11.494252 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493390    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493392    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493395    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493397    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493400    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493402    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493405    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493407    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493410    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493412    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493415    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493417    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493420    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493422    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493424    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493427    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493429    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493431    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493434    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:11.494718 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493436    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493439    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493442    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493444    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493446    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493449    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493452    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493456    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493458    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493461    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493463    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493466    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493468    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493471    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493473    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493476    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493478    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493481    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493483    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:11.495185 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493487    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493489    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493492    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493494    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493497    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493499    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.493502    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494300    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494308    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494311    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494314    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494316    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494319    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494322    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494325    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494328    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494331    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494334    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494336    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494339    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:11.495617 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494342    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494345    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494347    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494350    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494353    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494355    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494358    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494361    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494364    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494366    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494370    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494374    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494378    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494381    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494384    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494387    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494390    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494392    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494395    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:11.496244 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494397    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494400    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494402    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494405    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494408    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494410    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494413    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494416    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494418    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494422    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494425    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494427    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494430    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494433    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494435    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494438    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494440    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494443    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494445    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494448    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:11.496693 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494450    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494453    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494455    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494458    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494460    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494463    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494465    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494468    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494470    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494473    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494475    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494477    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494480    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494483    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494485    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494488    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494491    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494494    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494496    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494499    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:11.497175 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494501    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494504    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494508    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494510    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494513    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494515    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494517    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494520    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494523    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494525    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494527    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494530    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494533    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494535    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494538    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494540    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494543    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494545    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494547    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:11.497631 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494550    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494552    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494555    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494557    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494560    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494562    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494565    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494567    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494570    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494575    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494578    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494582    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494585    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494588    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.494590    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495099    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495106    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495109    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495112    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:11.498090 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495114    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495117    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495119    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495122    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495125    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495127    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495130    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495133    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495136    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495139    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495142    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495145    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495147    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495150    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495152    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495155    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495157    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495159    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495162    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:11.498552 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495169    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495171    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495174    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495177    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495179    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495182    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495185    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495188    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495190    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495193    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495196    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495199    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495202    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495204    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495207    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495209    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495212    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495214    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495216    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:11.499002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495219    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495222    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495225    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495227    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495231    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495235    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495238    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495240    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495242    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495245    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495248    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495250    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495252    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495256    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495259    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495261    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495264    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495266    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495269    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:11.499463 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495271    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495274    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495277    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495279    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495281    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495287    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495290    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495292    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495295    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495297    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495299    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495302    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495304    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495307    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495310    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495312    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495314    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495317    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495319    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495321    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:11.499904 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495324    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495326    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495329    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495331    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495334    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495336    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495340    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495343    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495345    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495348    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495350    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495353    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495355    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495358    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495360    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495363    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495365    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495368    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495371    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495374    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:11.500378 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495376    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495379    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495381    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495385    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.495388    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496194    2574 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496204    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496210    2574 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496214    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496219    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496222    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496229    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496233    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496236    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496239    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496243    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496246    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496250    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496253    2574 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496256    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496259    2574 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496262    2574 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496265    2574 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:24:11.500852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496270    2574 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496273    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496276    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496279    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496282    2574 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496288    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496291    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496293    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496297    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496300    2574 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496304    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496306    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496309    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496312    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496317    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496320    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496322    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496325    2574 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496328    2574 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496331    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496335    2574 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496338    2574 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496341    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496344    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:24:11.501453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496347    2574 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496351    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496353    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496357    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496360    2574 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496363    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496366    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496369    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496372    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496374    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496377    2574 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496380    2574 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496384    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496389    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496392    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496395    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496398    2574 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496402    2574 flags.go:64] FLAG: --help="false"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496405    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496408    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496411    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496415    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496418    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:24:11.502020 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496422    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496424    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496427    2574 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496430    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496433    2574 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496436    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496439    2574 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496442    2574 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496445    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496448    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496450    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496453    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496456    2574 flags.go:64] FLAG: --lock-file=""
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496459    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496462    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496465    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496470    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496473    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496476    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496478    2574 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496481    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496484    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496488    2574 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496491    2574 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496496    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:24:11.502599 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496499    2574 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496502    2574 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496506    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496509    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496512    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496515    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496518    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496520    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496523    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496530    2574 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496533    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496535    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496539    2574 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496541    2574 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496544    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496547    2574 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496550    2574 flags.go:64] FLAG: --port="10250"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496553    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496555    2574 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01e69977d1c5fe041"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496559    2574 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496562    2574 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496565    2574 flags.go:64] FLAG: --register-node="true"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496567    2574 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496571    2574 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496574    2574 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496577    2574 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:24:11.503210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496580    2574 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496583    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496586    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496590    2574 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496593    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496596    2574 flags.go:64] FLAG: --runonce="false"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496599    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496602    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496605    2574 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496608    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496611    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496614    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496617    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496620    2574 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496623    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496626    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496629    2574 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496631    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496634    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496637    2574 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496640    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496645    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496648    2574 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496650    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496655    2574 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:24:11.503804 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496659    2574 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496661    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496664    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496667    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496670    2574 flags.go:64] FLAG: --v="2"
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496674    2574 flags.go:64] FLAG: --version="false"
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496678    2574 flags.go:64] FLAG: --vmodule=""
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496682    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.496685    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496821    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496827    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496830    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496833    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496836    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496838    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496840    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496843    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496846    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496849    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496852    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496854    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:11.504399 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496857    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496860    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496862    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496865    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496867    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496870    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496872    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496875    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496877    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496879    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496882    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496885    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496888    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496891    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496893    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496896    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496898    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496901    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496904    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496906    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:11.504913 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496908    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496912    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496915    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496918    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496921    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496923    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496926    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496929    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496931    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496934    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496937    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496939    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496942    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496944    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496946    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496949    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496951    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496954    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496956    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:11.505401 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496959    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496961    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496964    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496966    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496970    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496972    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496975    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496977    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496980    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496982    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496985    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496987    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496990    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496993    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496995    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.496998    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497000    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497003    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497005    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497008    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:11.505843 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497010    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497014    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497018    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497021    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497024    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497027    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497030    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497032    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497035    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497038    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497040    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497043    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497045    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497047    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497050    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497052    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497056    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497059    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497061    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:11.506322 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497064    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497066    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497069    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497084    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497087    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497090    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497092    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497095    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497097    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497100    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497102    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497105    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497107    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497110    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497112    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497115    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:11.506767 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.497143    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497183    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497188    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497192    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497196    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497200    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497203    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497206    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497208    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497211    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497214    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497217    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497219    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497222    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497227    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497229    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497232    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497235    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497237    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:11.507186 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497240    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497242    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497245    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497247    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497250    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497253    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497255    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497258    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497260    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497263    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497265    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497268    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497270    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497273    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497275    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497278    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497281    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497283    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497286    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497288    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:11.507629 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497290    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497293    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497296    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497298    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497302    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497305    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497308    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497313    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497316    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497319    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497322    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497325    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497328    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497331    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497334    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497338    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497341    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497343    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497346    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:11.508095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497348    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497351    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497353    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497356    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497358    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497361    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497363    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497366    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497368    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497371    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497374    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497376    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497378    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497381    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497383    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497386    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497388    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497391    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497393    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497395    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:11.508539 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497400    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497403    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497405    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497408    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497410    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497413    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497415    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497418    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497420    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497423    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497425    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497428    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497431    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497433    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497436    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497438    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497441    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497443    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497446    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497448    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:11.509002 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497451    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497454    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497457    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497460    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497462    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497465    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497467    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497470    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.497472    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.504287    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.504301    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.504317    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.504322    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504391    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504396    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504399    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504403    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504406    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504408    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504411    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:11.509477 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504414    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504416    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504419    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504421    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504424    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504426    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504429    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504432    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504434    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504437    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504439    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504442    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504444    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504447    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504455    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504457    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504460    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504462    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504465    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504467    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:11.509957 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504470    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504472    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504480    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504483    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504486    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504489    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504492    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504494    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504497    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504500    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504504    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504507    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504510    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504513    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504515    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504518    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504520    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504523    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504525    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504528    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:11.510432 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504530    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504532    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504535    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504537    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504540    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504542    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504544    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504554    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504558    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504560    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504563    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504565    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504568    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504571    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504574    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504576    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504579    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504581    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504584    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504587    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:11.510936 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504589    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504593    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504596    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504599    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504601    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504604    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504607    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504609    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504611    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504614    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504616    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504619    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504621    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504623    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504626    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504628    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504631    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504633    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504636    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504638    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:11.511727 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504646    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504649    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504652    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504654    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504656    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504659    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504662    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504664    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504667    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504669    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504672    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504674    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504677    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504679    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504681    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504684    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504686    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504689    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:11.512462 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504691    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.504717    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504892    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504897    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504899    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504902    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504905    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504908    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504911    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504913    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504916    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504919    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504922    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504924    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504927    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504936    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:11.513070 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504939    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504941    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504944    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504946    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504949    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504951    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504954    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504956    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504959    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504962    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504966    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504969    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504972    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504975    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504978    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504980    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504983    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504985    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:11.513470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504988    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504990    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504993    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504995    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.504997    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505000    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505002    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505005    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505007    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505010    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505013    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505016    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505018    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505021    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505029    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505032    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505034    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505037    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505039    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505041    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:11.513956 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505044    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505046    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505048    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505051    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505053    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505056    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505058    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505060    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505063    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505065    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505068    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505070    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505091    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505095    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505098    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505101    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505104    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505106    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505109    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:11.514450 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505111    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505114    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505116    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505119    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505121    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505124    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505126    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505128    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505137    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505140    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505142    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505145    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505147    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505150    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505152    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505155    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505157    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505159    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505162    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505164    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:11.514971 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505167    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505170    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505172    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505175    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505177    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505180    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505182    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505185    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505187    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505189    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505192    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505194    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505197    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505199    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505202    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.505226    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:11.515561 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505269    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505273    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505276    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505279    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505282    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505290    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505293    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505296    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505298    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505300    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505303    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505305    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505308    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505310    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505312    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505315    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505317    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505319    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505322    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505324    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:11.515984 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505327    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505330    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505332    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505334    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505337    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505339    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505342    2574 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505344    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505346    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505349    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505351    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505353    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505356    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505358    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505361    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505363    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505365    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505368    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505375    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:11.516527 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505378    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505387    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505390    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505392    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505395    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505397    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505400    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505402    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505405    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505408    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505410    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505413    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505416    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505418    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505422    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505426    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505428    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505431    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505434    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:11.516963 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505436    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505438    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505441    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505443    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505445    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505450    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505453    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505455    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505458    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505460    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505463    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505465    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505468    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505476    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505478    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505481    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505484    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505486    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505489    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505492    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:11.517470 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505494    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505496    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505499    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505502    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505504    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505507    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505509    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505511    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505514    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505516    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505519    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505521    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505524    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505526    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505528    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505531    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505538    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505543    2574 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505546    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505549    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:11.517978 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505552    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505554    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505556    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505559    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505561    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505566    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505573    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:11.505576    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.505630    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.507447    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.508282    2574 server.go:1014] "Starting client certificate rotation"
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.508393    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.508439    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.511783    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:24:11.518597 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.514776    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:24:11.522560 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.522542    2574 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:24:11.525265 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.525251    2574 log.go:25] "Validated CRI v1 image API"
Nov 02 01:24:11.525989 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.525968    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:24:11.528255 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.528228    2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 95cbff6b-6ab0-43bf-8329-a3b10046fe6e:/dev/nvme0n1p4 b70a65a0-5ff9-41c8-bcac-70b5661b8df4:/dev/nvme0n1p3]
Nov 02 01:24:11.528315 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.528254    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 01:24:11.533224 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.533116    2574 manager.go:217] Machine: {Timestamp:2026-11-02 01:24:11.53185718 +0000 UTC m=+0.286412029 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103923 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b202a86ac308c2a1f90c19d486ab5 SystemUUID:ec2b202a-86ac-308c-2a1f-90c19d486ab5 BootID:6aa6274f-7055-4b5f-ac37-69db31cb4479 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:64:ef:25:82:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:64:ef:25:82:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:c1:63:7c:4f:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:24:11.533617 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.533608    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:24:11.533725 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.533713    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:24:11.536196 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.536170    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:24:11.536326 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.536198    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-183.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:24:11.536366 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.536333    2574 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:24:11.536366 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.536341    2574 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:24:11.536419 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.536370    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:24:11.536419 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.536383    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:24:11.536996 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.536987    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:24:11.540624 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.540612    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:24:11.540739 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.540730    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:24:11.541358 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.541349    2574 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:24:11.541388 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.541363    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:24:11.541388 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.541374    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:24:11.541388 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.541384    2574 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:24:11.541470 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.541396    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:24:11.541801 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.541776    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:24:11.543953 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.543938    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:24:11.548178 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.548162    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:24:11.548838 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.548826    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:24:11.548886 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.548856    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:24:11.549009 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.548999    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:24:11.549044 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549015    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:24:11.549044 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549021    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:24:11.549044 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549027    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:24:11.549044 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549032    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:24:11.549044 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549037    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:24:11.549875 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549866    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:24:11.549904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549875    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:24:11.549904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549883    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:24:11.549904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549890    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:24:11.549973 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549906    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:24:11.549973 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.549915    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:24:11.550398 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.550387    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:24:11.550398 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.550398    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:24:11.554180 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.554162    2574 server.go:1263] "Started kubelet"
Nov 02 01:24:11.554258 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.554219    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:24:11.554323 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.554259    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:24:11.554376 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.554341    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:24:11.554969 ip-10-0-134-183 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:24:11.555463 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.555444    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:24:11.556504 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.556486    2574 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:24:11.560282 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.560266    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:24:11.560613 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.560589    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:24:11.560988 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.560962    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-183.us-west-1.compute.internal\" not found"
Nov 02 01:24:11.561498 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.561175    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:24:11.561498 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.561189    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:24:11.561498 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.561344    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:24:11.561498 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.561360    2574 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:24:11.561498 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.561371    2574 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:24:11.562037 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.562017    2574 factory.go:55] Registering systemd factory
Nov 02 01:24:11.562037 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.562037    2574 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:24:11.562310 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.562294    2574 factory.go:153] Registering CRI-O factory
Nov 02 01:24:11.562369 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.562314    2574 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:24:11.562399 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.562388    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:24:11.562440 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.562410    2574 factory.go:103] Registering Raw factory
Nov 02 01:24:11.562440 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.562425    2574 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:24:11.562764 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.562712    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 01:24:11.563368 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.563292    2574 manager.go:319] Starting recovery of all containers
Nov 02 01:24:11.574120 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.574099    2574 manager.go:324] Recovery completed
Nov 02 01:24:11.579151 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.579136    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:11.580939 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.580914    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:11.580998 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.580952    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:11.580998 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.580966    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:11.581484 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.581471    2574 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:24:11.581543 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.581484    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:24:11.581543 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.581476    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:11.581543 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.581504    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:24:11.582107 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.582092    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:11.582169 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.582123    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:11.582169 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.582136    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:11.583708 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.583693    2574 policy_none.go:50] "Start"
Nov 02 01:24:11.583781 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.583712    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:24:11.583781 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.583724    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:24:11.585738 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.585727    2574 policy_none.go:44] "Start"
Nov 02 01:24:11.620648 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.620614    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:24:11.622921 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.622909    2574 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:24:11.623313 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.623295    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:24:11.623357 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.623316    2574 server.go:87] "Starting device plugin registration server"
Nov 02 01:24:11.623576 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.623565    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:24:11.623623 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.623582    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:24:11.623717 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.623700    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:24:11.623814 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.623790    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:24:11.623814 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.623801    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:24:11.624250 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.624230    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:24:11.624313 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.624291    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-183.us-west-1.compute.internal\" not found"
Nov 02 01:24:11.696908 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.696883    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:24:11.698977 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.698268    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:24:11.698977 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.698282    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:24:11.698977 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.698297    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:24:11.698977 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.698327    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 01:24:11.705857 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.705834    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-183.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 01:24:11.705857 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.705838    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-183.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 01:24:11.706793 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.705841    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-183.us-west-1.compute.internal.18e415eb352fffd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-183.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-183.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:24:11.554136022 +0000 UTC m=+0.308690871,LastTimestamp:2026-11-02 01:24:11.554136022 +0000 UTC m=+0.308690871,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-183.us-west-1.compute.internal,}"
Nov 02 01:24:11.724393 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.724377    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:11.725105 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.725071    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:11.725179 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.725117    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:11.725179 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.725128    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:11.725179 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.725148    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.788296 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.788275    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:24:11.788366 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.788313    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qbthf"
Nov 02 01:24:11.799315 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.799295    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"]
Nov 02 01:24:11.799364 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.799347    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:11.800724 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.800710    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:11.800782 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.800738    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:11.800782 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.800747    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:11.802948 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.802937    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:11.803114 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.803098    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.803153 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.803129    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:11.803852 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.803839    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:11.803900 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.803865    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:11.803900 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.803874    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:11.803963 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.803840    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:11.803963 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.803937    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:11.803963 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.803946    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:11.806367 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.806355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.806420 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.806379    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:11.807122 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.807105    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:11.807189 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.807140    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:11.807189 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.807171    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:11.831400 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.831383    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-183.us-west-1.compute.internal\" not found" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.834435 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.834418    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-183.us-west-1.compute.internal\" not found" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.864115 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.864099    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:11.865417 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.865403    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:11.962497 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.962478    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:11.963048 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.963026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.963120 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.963057    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.963120 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.963099    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c8e4877a18ae50f2a48041b0698e786-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"8c8e4877a18ae50f2a48041b0698e786\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.965243 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.965131    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:11.965243 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:11.965162    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-183.us-west-1.compute.internal\": node \"ip-10-0-134-183.us-west-1.compute.internal\" not found"
Nov 02 01:24:11.965478 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:11.965454    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:12.061210 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.061181    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.063392 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.063371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.063507 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.063396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.063507 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.063413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c8e4877a18ae50f2a48041b0698e786-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"8c8e4877a18ae50f2a48041b0698e786\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.063507 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.063458    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c8e4877a18ae50f2a48041b0698e786-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"8c8e4877a18ae50f2a48041b0698e786\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.063507 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.063468    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.063507 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.063477    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.121644 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.121628    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:24:12.121739 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.121722    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.134303 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.134285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.137179 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.137155    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:12.180065 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.180046    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:24:12.508504 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.508438    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 01:24:12.508951 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.508573    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:24:12.508951 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.508587    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:24:12.508951 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.508608    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 01:24:12.508951 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:12.508556    2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a379cc44211674ec397976e30fa324b1-fb94896701b40ffa.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-183.us-west-1.compute.internal.18e415eb36c97308\": read tcp 10.0.134.183:42572->3.20.245.63:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-183.us-west-1.compute.internal.18e415eb36c97308  default   5058 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-183.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-183.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-183.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:24:11 +0000 UTC,LastTimestamp:2026-11-02 01:24:11.725131912 +0000 UTC m=+0.479686761,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-183.us-west-1.compute.internal,}"
Nov 02 01:24:12.542328 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.542313    2574 apiserver.go:52] "Watching apiserver"
Nov 02 01:24:12.561034 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.561018    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 01:24:12.793112 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.793069    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:19:12 +0000 UTC" deadline="2028-04-16 03:44:40.131849312 +0000 UTC"
Nov 02 01:24:12.793112 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.793108    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12746h20m27.338743592s"
Nov 02 01:24:12.817720 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.817692    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:24:12.818026 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.818008    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-26vc2","openshift-network-diagnostics/network-check-target-wjrcp","openshift-network-operator/iptables-alerter-sf489","openshift-ovn-kubernetes/ovnkube-node-kjszn","kube-system/global-pull-secret-syncer-hcbh5","kube-system/konnectivity-agent-vvj9k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj","openshift-image-registry/node-ca-9pdx5","openshift-cluster-node-tuning-operator/tuned-7pbwr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-cdbt2","openshift-multus/multus-kbt65","kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal","openshift-dns/node-resolver-q77dv"]
Nov 02 01:24:12.820390 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.820374    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:12.820453 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:12.820437    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:12.822378 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.822366    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:12.822421 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:12.822412    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:12.824365 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.824348    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.826576 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.826558    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.828606 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.828590    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.828670 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:12.828640    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:12.830812 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.830798    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:12.832947 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.832930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.835307 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.835186    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.837324 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.837308    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.840018 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.840000    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.842328 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.842312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.844439 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.844425    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.862065 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.862050    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:24:12.866586 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-systemd-units\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.866586 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-script-lib\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.866689 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvvr8\" (UniqueName: \"kubernetes.io/projected/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-kube-api-access-tvvr8\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.866689 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-k8s-cni-cncf-io\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.866689 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-env-overrides\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.866689 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-socket-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.866816 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866717    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-kubernetes\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.866816 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-conf\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.866816 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-cni-binary-copy\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.866816 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-multus\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.866929 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866819    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-etc-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.866929 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-bin\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.866929 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-netd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.866929 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77f27\" (UniqueName: \"kubernetes.io/projected/0406fb90-83ff-40c9-a3aa-e834abf84e46-kube-api-access-77f27\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.866929 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.867110 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866980    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.867110 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.866996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-node-log\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867110 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-var-lib-kubelet\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.867110 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2j7d\" (UniqueName: \"kubernetes.io/projected/55451071-26c3-4ebd-9377-d5cc1b967ea5-kube-api-access-c2j7d\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:12.867110 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:12.867110 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bc310c2-330e-492f-b635-5d689f1e5203-host\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.867110 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-netns\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-kubelet\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867163    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba2de926-a683-49c8-9d4d-f45a09702479-iptables-alerter-script\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867182    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-lib-modules\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867257    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-host\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxns9\" (UniqueName: \"kubernetes.io/projected/ba2de926-a683-49c8-9d4d-f45a09702479-kube-api-access-pxns9\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-device-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.867309 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867298    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-run\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-binary-copy\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f612b3f-b624-45ae-b05d-2b4a164fd4af-agent-certs\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-cnibin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txddj\" (UniqueName: \"kubernetes.io/projected/87856a71-2d8f-47f8-8d85-4d797fb8bec4-kube-api-access-txddj\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-slash\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-ovn\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-system-cni-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-dbus\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867472    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-system-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867495    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-netns\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.867549 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867545    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-socket-dir-parent\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-bin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-etc-kubernetes\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867674    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-conf-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867688    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-kubelet\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-config\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovn-node-metrics-cert\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867793    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-modprobe-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f612b3f-b624-45ae-b05d-2b4a164fd4af-konnectivity-ca\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7chh\" (UniqueName: \"kubernetes.io/projected/3464bf51-0036-4948-8bb4-43f1000d58d2-kube-api-access-l7chh\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.867884 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjqms\" (UniqueName: \"kubernetes.io/projected/6bc310c2-330e-492f-b635-5d689f1e5203-kube-api-access-vjqms\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867907    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-systemd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-var-lib-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-tuned\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-tmp\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.867990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-kubelet-config\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868007    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-hostroot\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-multus-certs\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87856a71-2d8f-47f8-8d85-4d797fb8bec4-hosts-file\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868069    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-log-socket\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-sys-fs\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868147    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bc310c2-330e-492f-b635-5d689f1e5203-serviceca\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvq2v\" (UniqueName: \"kubernetes.io/projected/9766361f-a2b3-4c58-9f47-899e699c2520-kube-api-access-bvq2v\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868229    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87856a71-2d8f-47f8-8d85-4d797fb8bec4-tmp-dir\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868249    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2de926-a683-49c8-9d4d-f45a09702479-host-slash\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysconfig\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.868451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-sys\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.868898 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868292    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-cnibin\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.868898 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-multus-daemon-config\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.868898 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-registration-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.868898 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-systemd\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.868898 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.868355    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhmx\" (UniqueName: \"kubernetes.io/projected/46964574-0965-4bf3-8ce7-1c3c5115c594-kube-api-access-pwhmx\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.876311 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.876288    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:24:12.876393 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.876327    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:24:12.876393 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.876335    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kv2jl\""
Nov 02 01:24:12.876393 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.876331    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:24:12.878287 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.878271    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:24:12.878287 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.878274    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:24:12.878386 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.878293    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:24:12.878386 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.878325    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:24:12.878451 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.878411    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:24:12.878566 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.878553    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:24:12.878566 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.878562    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pw4q8\""
Nov 02 01:24:12.882215 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.882199    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:24:12.882362 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.882348    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:24:12.882582 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.882570    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gh69v\""
Nov 02 01:24:12.884491 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.884478    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:24:12.919590 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.919571    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:24:12.919700 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.919688    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:24:12.920266 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.920216    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5zpq\""
Nov 02 01:24:12.920364 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.920317    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2dvkb\""
Nov 02 01:24:12.920364 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.920320    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:24:12.920364 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.920332    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wwr4r\""
Nov 02 01:24:12.920364 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.920332    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gsg88\""
Nov 02 01:24:12.921582 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.921564    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwsn\""
Nov 02 01:24:12.921582 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.921575    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:24:12.921582 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.921574    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfbzc\""
Nov 02 01:24:12.921706 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.921580    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:24:12.921706 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.921610    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 01:24:12.921706 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.921641    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xh9v4"
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.925990    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926006    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926027    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926045    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926049    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926054    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926048    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926109    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926113    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:24:12.926136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.926136    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:24:12.968579 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-systemd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.968579 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-var-lib-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.968685 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-tuned\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.968685 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-tmp\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.968685 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-kubelet-config\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.968685 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-hostroot\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.968685 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968674    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-systemd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-multus-certs\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968709    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-var-lib-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87856a71-2d8f-47f8-8d85-4d797fb8bec4-hosts-file\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-log-socket\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968756    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-multus-certs\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968767    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-sys-fs\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968798    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-log-socket\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87856a71-2d8f-47f8-8d85-4d797fb8bec4-hosts-file\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bc310c2-330e-492f-b635-5d689f1e5203-serviceca\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968826    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-hostroot\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968841    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-sys-fs\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvq2v\" (UniqueName: \"kubernetes.io/projected/9766361f-a2b3-4c58-9f47-899e699c2520-kube-api-access-bvq2v\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87856a71-2d8f-47f8-8d85-4d797fb8bec4-tmp-dir\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968860    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-kubelet-config\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.968904 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:12.968897    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968940    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2de926-a683-49c8-9d4d-f45a09702479-host-slash\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:12.968987    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs podName:55451071-26c3-4ebd-9377-d5cc1b967ea5 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:13.468964089 +0000 UTC m=+2.223519013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs") pod "network-metrics-daemon-26vc2" (UID: "55451071-26c3-4ebd-9377-d5cc1b967ea5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.968998    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2de926-a683-49c8-9d4d-f45a09702479-host-slash\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969007    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysconfig\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-sys\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-cnibin\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-multus-daemon-config\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-registration-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-systemd\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969159    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-sys\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969176    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhmx\" (UniqueName: \"kubernetes.io/projected/46964574-0965-4bf3-8ce7-1c3c5115c594-kube-api-access-pwhmx\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-systemd-units\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysconfig\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969246    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-script-lib\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969259    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-systemd-units\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.969705 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969284    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvr8\" (UniqueName: \"kubernetes.io/projected/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-kube-api-access-tvvr8\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969262    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87856a71-2d8f-47f8-8d85-4d797fb8bec4-tmp-dir\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-k8s-cni-cncf-io\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969320    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bc310c2-330e-492f-b635-5d689f1e5203-serviceca\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-env-overrides\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969251    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-registration-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969309    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-systemd\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-socket-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-kubernetes\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-conf\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-cni-binary-copy\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969522    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-multus\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969286    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-cnibin\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969565    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-kubernetes\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969611    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-conf\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-etc-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969686    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-etc-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969702    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-bin\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.970559 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-netd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77f27\" (UniqueName: \"kubernetes.io/projected/0406fb90-83ff-40c9-a3aa-e834abf84e46-kube-api-access-77f27\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969801    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-env-overrides\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969809    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969815    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-k8s-cni-cncf-io\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-node-log\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969786    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-socket-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969852    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-netd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969888    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-multus-daemon-config\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-var-lib-kubelet\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2j7d\" (UniqueName: \"kubernetes.io/projected/55451071-26c3-4ebd-9377-d5cc1b967ea5-kube-api-access-c2j7d\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970194    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-script-lib\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970010    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-node-log\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.969978    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-cni-binary-copy\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.971295 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bc310c2-330e-492f-b635-5d689f1e5203-host\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970417    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-netns\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970437    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-multus\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-bin\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970531    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-kubelet\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba2de926-a683-49c8-9d4d-f45a09702479-iptables-alerter-script\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970668    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bc310c2-330e-492f-b635-5d689f1e5203-host\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970730    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-kubelet\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-lib-modules\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-host\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970780    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-var-lib-kubelet\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxns9\" (UniqueName: \"kubernetes.io/projected/ba2de926-a683-49c8-9d4d-f45a09702479-kube-api-access-pxns9\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-device-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-run\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.972116 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-binary-copy\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970900    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-lib-modules\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970915    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f612b3f-b624-45ae-b05d-2b4a164fd4af-agent-certs\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970944    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-cnibin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txddj\" (UniqueName: \"kubernetes.io/projected/87856a71-2d8f-47f8-8d85-4d797fb8bec4-kube-api-access-txddj\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-slash\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-ovn\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971110    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-host\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971110    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-system-cni-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971153    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-system-cni-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-dbus\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-system-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971198    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-dbus\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971242    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba2de926-a683-49c8-9d4d-f45a09702479-iptables-alerter-script\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971292    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-slash\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.970816    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-netns\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.972930 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:12.971358    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:12.971402    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret podName:1955308f-bad5-47ef-951a-95b7064ed5b2 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:13.471386898 +0000 UTC m=+2.225941756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret") pod "global-pull-secret-syncer-hcbh5" (UID: "1955308f-bad5-47ef-951a-95b7064ed5b2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971038    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971420    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-run\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-netns\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971456    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-ovn\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971499    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-device-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-socket-dir-parent\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-bin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971559    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-netns\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-etc-kubernetes\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971616    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-cnibin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971616    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-system-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971633    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-binary-copy\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.973604 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971653    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-etc-kubernetes\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971682    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971673    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-socket-dir-parent\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971720    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-bin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971768    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-conf-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-kubelet\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971837    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971861    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-config\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-conf-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.971934    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-kubelet\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovn-node-metrics-cert\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-modprobe-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f612b3f-b624-45ae-b05d-2b4a164fd4af-konnectivity-ca\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972230    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.974058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7chh\" (UniqueName: \"kubernetes.io/projected/3464bf51-0036-4948-8bb4-43f1000d58d2-kube-api-access-l7chh\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972310    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqms\" (UniqueName: \"kubernetes.io/projected/6bc310c2-330e-492f-b635-5d689f1e5203-kube-api-access-vjqms\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972566    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972565    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972639    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-modprobe-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972729    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-config\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.972794    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.973283    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-tuned\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.973338    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-tmp\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.973450    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f612b3f-b624-45ae-b05d-2b4a164fd4af-konnectivity-ca\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:12.974541 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.974253    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f612b3f-b624-45ae-b05d-2b4a164fd4af-agent-certs\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:12.975275 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:12.975256    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovn-node-metrics-cert\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:13.027751 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.027712    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:13.027751 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.027743    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:13.027924 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.027759    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pwtcv for pod openshift-network-diagnostics/network-check-target-wjrcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:13.027924 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.027827    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv podName:e7237264-dbfe-4566-8e42-8ddc59360817 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:13.527808145 +0000 UTC m=+2.282363003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pwtcv" (UniqueName: "kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv") pod "network-check-target-wjrcp" (UID: "e7237264-dbfe-4566-8e42-8ddc59360817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:13.030140 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.030112    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhmx\" (UniqueName: \"kubernetes.io/projected/46964574-0965-4bf3-8ce7-1c3c5115c594-kube-api-access-pwhmx\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:13.030249 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.030144    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvr8\" (UniqueName: \"kubernetes.io/projected/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-kube-api-access-tvvr8\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:13.030249 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.030121    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvq2v\" (UniqueName: \"kubernetes.io/projected/9766361f-a2b3-4c58-9f47-899e699c2520-kube-api-access-bvq2v\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:13.031296 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.031270    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2j7d\" (UniqueName: \"kubernetes.io/projected/55451071-26c3-4ebd-9377-d5cc1b967ea5-kube-api-access-c2j7d\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:13.031980 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.031957    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxns9\" (UniqueName: \"kubernetes.io/projected/ba2de926-a683-49c8-9d4d-f45a09702479-kube-api-access-pxns9\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:13.032318 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.032297    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqms\" (UniqueName: \"kubernetes.io/projected/6bc310c2-330e-492f-b635-5d689f1e5203-kube-api-access-vjqms\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:13.032447 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.032414    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txddj\" (UniqueName: \"kubernetes.io/projected/87856a71-2d8f-47f8-8d85-4d797fb8bec4-kube-api-access-txddj\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:13.032523 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.032509    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77f27\" (UniqueName: \"kubernetes.io/projected/0406fb90-83ff-40c9-a3aa-e834abf84e46-kube-api-access-77f27\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:13.032702 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.032687    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7chh\" (UniqueName: \"kubernetes.io/projected/3464bf51-0036-4948-8bb4-43f1000d58d2-kube-api-access-l7chh\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:13.155365 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.155283    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:13.175601 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.175582    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:13.178174 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.178138    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0483b9f053a944bf50e838e170a69c85.slice/crio-9348b2b7de5c8002e9acccaa25ccd54b4a9c0aa73e50d889c3d5e39f1726b304 WatchSource:0}: Error finding container 9348b2b7de5c8002e9acccaa25ccd54b4a9c0aa73e50d889c3d5e39f1726b304: Status 404 returned error can't find the container with id 9348b2b7de5c8002e9acccaa25ccd54b4a9c0aa73e50d889c3d5e39f1726b304
Nov 02 01:24:13.179129 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.179104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:13.180369 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.180342    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2de926_a683_49c8_9d4d_f45a09702479.slice/crio-0048a8958faf75002bcbc53d3b5affbd0bb3cdece1237372bc495e39b454dd8e WatchSource:0}: Error finding container 0048a8958faf75002bcbc53d3b5affbd0bb3cdece1237372bc495e39b454dd8e: Status 404 returned error can't find the container with id 0048a8958faf75002bcbc53d3b5affbd0bb3cdece1237372bc495e39b454dd8e
Nov 02 01:24:13.181095 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.181061    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8e4877a18ae50f2a48041b0698e786.slice/crio-0b0586fd638a3ee62d41a31d7810c079d0a958dd8976fd2fda174951f4399d34 WatchSource:0}: Error finding container 0b0586fd638a3ee62d41a31d7810c079d0a958dd8976fd2fda174951f4399d34: Status 404 returned error can't find the container with id 0b0586fd638a3ee62d41a31d7810c079d0a958dd8976fd2fda174951f4399d34
Nov 02 01:24:13.185321 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.185306    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:24:13.186810 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.186741    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0406fb90_83ff_40c9_a3aa_e834abf84e46.slice/crio-029b8e4b89efc967f55fb182cfc26eb92d8e669fa4d0685c46ccd65b850d1543 WatchSource:0}: Error finding container 029b8e4b89efc967f55fb182cfc26eb92d8e669fa4d0685c46ccd65b850d1543: Status 404 returned error can't find the container with id 029b8e4b89efc967f55fb182cfc26eb92d8e669fa4d0685c46ccd65b850d1543
Nov 02 01:24:13.189433 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.189416    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f612b3f_b624_45ae_b05d_2b4a164fd4af.slice/crio-e6431d4e9b3a09c4ae90e1b5e9fcad8a814c8c87103c0094892e5c955729f9b9 WatchSource:0}: Error finding container e6431d4e9b3a09c4ae90e1b5e9fcad8a814c8c87103c0094892e5c955729f9b9: Status 404 returned error can't find the container with id e6431d4e9b3a09c4ae90e1b5e9fcad8a814c8c87103c0094892e5c955729f9b9
Nov 02 01:24:13.217810 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.217787    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:13.217881 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.217793    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:13.226686 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.226670    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3464bf51_0036_4948_8bb4_43f1000d58d2.slice/crio-b121f4d54e4477af976f7a35e0fbad8412c0f6208dada04f44275e5fda0c9161 WatchSource:0}: Error finding container b121f4d54e4477af976f7a35e0fbad8412c0f6208dada04f44275e5fda0c9161: Status 404 returned error can't find the container with id b121f4d54e4477af976f7a35e0fbad8412c0f6208dada04f44275e5fda0c9161
Nov 02 01:24:13.228633 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.228592    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc310c2_330e_492f_b635_5d689f1e5203.slice/crio-1501fcf156f5fa7938f47025ea1a96be8045f2b27bd3cb56979230a4be6f035c WatchSource:0}: Error finding container 1501fcf156f5fa7938f47025ea1a96be8045f2b27bd3cb56979230a4be6f035c: Status 404 returned error can't find the container with id 1501fcf156f5fa7938f47025ea1a96be8045f2b27bd3cb56979230a4be6f035c
Nov 02 01:24:13.245648 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.245633    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:13.253722 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.253702    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d20a7d0_d504_4b4d_89a6_2c48b5de6315.slice/crio-3db30c2760fe3e9957717e1f34e79178acebf8d90bcc3ea9e291b9fd2ef54751 WatchSource:0}: Error finding container 3db30c2760fe3e9957717e1f34e79178acebf8d90bcc3ea9e291b9fd2ef54751: Status 404 returned error can't find the container with id 3db30c2760fe3e9957717e1f34e79178acebf8d90bcc3ea9e291b9fd2ef54751
Nov 02 01:24:13.273311 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.273296    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:13.276606 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.276592    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kbt65"
Nov 02 01:24:13.282733 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.282707    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46964574_0965_4bf3_8ce7_1c3c5115c594.slice/crio-c3d712a65735714262878f625e57f91c8b7a1061b2e48d852a96b2b9f974cdba WatchSource:0}: Error finding container c3d712a65735714262878f625e57f91c8b7a1061b2e48d852a96b2b9f974cdba: Status 404 returned error can't find the container with id c3d712a65735714262878f625e57f91c8b7a1061b2e48d852a96b2b9f974cdba
Nov 02 01:24:13.285260 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.285242    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:13.285472 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.285457    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9766361f_a2b3_4c58_9f47_899e699c2520.slice/crio-f7068729c8fb9d6c6b2e7d33c9b580b585d98a289cb68f83ed916dcc5452867c WatchSource:0}: Error finding container f7068729c8fb9d6c6b2e7d33c9b580b585d98a289cb68f83ed916dcc5452867c: Status 404 returned error can't find the container with id f7068729c8fb9d6c6b2e7d33c9b580b585d98a289cb68f83ed916dcc5452867c
Nov 02 01:24:13.293611 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:13.293593    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87856a71_2d8f_47f8_8d85_4d797fb8bec4.slice/crio-a0fdcd2d101d0950f75b48340425da91c9b4b6c529ab59b8f8e4f0540948f95b WatchSource:0}: Error finding container a0fdcd2d101d0950f75b48340425da91c9b4b6c529ab59b8f8e4f0540948f95b: Status 404 returned error can't find the container with id a0fdcd2d101d0950f75b48340425da91c9b4b6c529ab59b8f8e4f0540948f95b
Nov 02 01:24:13.295255 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.295237    2574 kuberuntime_manager.go:1707] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-q77dv_openshift-dns(87856a71-2d8f-47f8-8d85-4d797fb8bec4): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Nov 02 01:24:13.295337 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.295264    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-q77dv" podUID="87856a71-2d8f-47f8-8d85-4d797fb8bec4"
Nov 02 01:24:13.476316 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.476245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:13.476316 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.476288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:13.476482 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.476406    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:13.476482 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.476406    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:13.476482 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.476461    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret podName:1955308f-bad5-47ef-951a-95b7064ed5b2 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:14.476444254 +0000 UTC m=+3.230999091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret") pod "global-pull-secret-syncer-hcbh5" (UID: "1955308f-bad5-47ef-951a-95b7064ed5b2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:13.476482 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.476474    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs podName:55451071-26c3-4ebd-9377-d5cc1b967ea5 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:14.476467761 +0000 UTC m=+3.231022596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs") pod "network-metrics-daemon-26vc2" (UID: "55451071-26c3-4ebd-9377-d5cc1b967ea5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:13.576786 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:13.576758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:13.577270 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.576853    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:13.577270 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.576865    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:13.577270 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.576873    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pwtcv for pod openshift-network-diagnostics/network-check-target-wjrcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:13.577270 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:13.576911    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv podName:e7237264-dbfe-4566-8e42-8ddc59360817 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:14.576900244 +0000 UTC m=+3.331455080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwtcv" (UniqueName: "kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv") pod "network-check-target-wjrcp" (UID: "e7237264-dbfe-4566-8e42-8ddc59360817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:14.908480 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.908465    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:14.945709 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.945668    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q77dv" event={"ID":"87856a71-2d8f-47f8-8d85-4d797fb8bec4","Type":"ContainerStarted","Data":"a0fdcd2d101d0950f75b48340425da91c9b4b6c529ab59b8f8e4f0540948f95b"}
Nov 02 01:24:14.946800 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.946778    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerStarted","Data":"c3d712a65735714262878f625e57f91c8b7a1061b2e48d852a96b2b9f974cdba"}
Nov 02 01:24:14.947252 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:14.947228    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a8b58f028a9a4a0345ad1f809e47fb9913d7f77c731bfd785819e1c71be9472\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-q77dv" podUID="87856a71-2d8f-47f8-8d85-4d797fb8bec4"
Nov 02 01:24:14.947874 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.947857    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7pbwr" event={"ID":"7d20a7d0-d504-4b4d-89a6-2c48b5de6315","Type":"ContainerStarted","Data":"3db30c2760fe3e9957717e1f34e79178acebf8d90bcc3ea9e291b9fd2ef54751"}
Nov 02 01:24:14.948875 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.948857    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9pdx5" event={"ID":"6bc310c2-330e-492f-b635-5d689f1e5203","Type":"ContainerStarted","Data":"1501fcf156f5fa7938f47025ea1a96be8045f2b27bd3cb56979230a4be6f035c"}
Nov 02 01:24:14.949879 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.949862    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kbt65" event={"ID":"9766361f-a2b3-4c58-9f47-899e699c2520","Type":"ContainerStarted","Data":"f7068729c8fb9d6c6b2e7d33c9b580b585d98a289cb68f83ed916dcc5452867c"}
Nov 02 01:24:14.950854 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.950832    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj" event={"ID":"3464bf51-0036-4948-8bb4-43f1000d58d2","Type":"ContainerStarted","Data":"b121f4d54e4477af976f7a35e0fbad8412c0f6208dada04f44275e5fda0c9161"}
Nov 02 01:24:14.951717 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.951702    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvj9k" event={"ID":"3f612b3f-b624-45ae-b05d-2b4a164fd4af","Type":"ContainerStarted","Data":"e6431d4e9b3a09c4ae90e1b5e9fcad8a814c8c87103c0094892e5c955729f9b9"}
Nov 02 01:24:14.953123 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.953103    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"029b8e4b89efc967f55fb182cfc26eb92d8e669fa4d0685c46ccd65b850d1543"}
Nov 02 01:24:14.954023 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.954005    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"8c8e4877a18ae50f2a48041b0698e786","Type":"ContainerStarted","Data":"0b0586fd638a3ee62d41a31d7810c079d0a958dd8976fd2fda174951f4399d34"}
Nov 02 01:24:14.955058 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.955041    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sf489" event={"ID":"ba2de926-a683-49c8-9d4d-f45a09702479","Type":"ContainerStarted","Data":"0048a8958faf75002bcbc53d3b5affbd0bb3cdece1237372bc495e39b454dd8e"}
Nov 02 01:24:14.956012 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:14.955990    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"0483b9f053a944bf50e838e170a69c85","Type":"ContainerStarted","Data":"9348b2b7de5c8002e9acccaa25ccd54b4a9c0aa73e50d889c3d5e39f1726b304"}
Nov 02 01:24:15.167740 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.167694    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:13 +0000 UTC" deadline="2028-07-14 09:33:09.213856464 +0000 UTC"
Nov 02 01:24:15.167740 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.167735    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14888h8m54.046123567s"
Nov 02 01:24:15.167740 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.167739    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:15.280952 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.280922    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:15.726674 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.726589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:15.726674 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.726677    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:15.726969 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.726858    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:15.726969 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.726875    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:15.726969 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.726937    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs podName:55451071-26c3-4ebd-9377-d5cc1b967ea5 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:17.726901532 +0000 UTC m=+5.237925009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs") pod "network-metrics-daemon-26vc2" (UID: "55451071-26c3-4ebd-9377-d5cc1b967ea5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:15.726969 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.726958    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret podName:1955308f-bad5-47ef-951a-95b7064ed5b2 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:17.72694792 +0000 UTC m=+5.237971379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret") pod "global-pull-secret-syncer-hcbh5" (UID: "1955308f-bad5-47ef-951a-95b7064ed5b2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:15.831638 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.827498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:15.831638 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.827763    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:15.831638 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.827783    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:15.831638 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.827795    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pwtcv for pod openshift-network-diagnostics/network-check-target-wjrcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:15.831638 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.827853    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv podName:e7237264-dbfe-4566-8e42-8ddc59360817 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:17.827835713 +0000 UTC m=+5.338859174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwtcv" (UniqueName: "kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv") pod "network-check-target-wjrcp" (UID: "e7237264-dbfe-4566-8e42-8ddc59360817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:15.943699 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.943665    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:15.943875 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.943813    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:15.943937 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.943893    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:15.943937 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:15.943915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:15.944031 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.943992    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:15.944085 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:15.944063    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:16.168640 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:16.168559    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:13 +0000 UTC" deadline="2028-06-11 12:23:57.312454784 +0000 UTC"
Nov 02 01:24:16.168640 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:16.168637    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14098h59m41.143821443s"
Nov 02 01:24:17.742333 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:17.741856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:17.742333 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:17.741924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:17.742333 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.742104    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:17.742333 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.742168    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs podName:55451071-26c3-4ebd-9377-d5cc1b967ea5 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:21.742150525 +0000 UTC m=+9.253173996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs") pod "network-metrics-daemon-26vc2" (UID: "55451071-26c3-4ebd-9377-d5cc1b967ea5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:17.742333 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.742270    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:17.742333 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.742305    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret podName:1955308f-bad5-47ef-951a-95b7064ed5b2 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:21.742294148 +0000 UTC m=+9.253317616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret") pod "global-pull-secret-syncer-hcbh5" (UID: "1955308f-bad5-47ef-951a-95b7064ed5b2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:17.842357 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:17.842324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:17.842523 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.842460    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:17.842523 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.842475    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:17.842523 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.842486    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pwtcv for pod openshift-network-diagnostics/network-check-target-wjrcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:17.842697 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.842534    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv podName:e7237264-dbfe-4566-8e42-8ddc59360817 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:21.8425207 +0000 UTC m=+9.353544153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwtcv" (UniqueName: "kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv") pod "network-check-target-wjrcp" (UID: "e7237264-dbfe-4566-8e42-8ddc59360817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:17.942992 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:17.942464    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:17.942992 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.942601    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:17.942992 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:17.942717    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:17.942992 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.942809    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:17.942992 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:17.942853    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:17.942992 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:17.942919    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:19.001484 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.001240    2574 generic.go:356] "Generic (PLEG): container finished" podID="0483b9f053a944bf50e838e170a69c85" containerID="a6a1f0beda2fedfaaa5a0d63439bd3328d945e89969609091b85ba408bc5b072" exitCode=0
Nov 02 01:24:19.002122 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.001543    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"0483b9f053a944bf50e838e170a69c85","Type":"ContainerDied","Data":"a6a1f0beda2fedfaaa5a0d63439bd3328d945e89969609091b85ba408bc5b072"}
Nov 02 01:24:19.005930 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.005901    2574 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="a5d53eea0d5802e8c175bcf398b77ec81f3de2c2953fd80064066594285391ee" exitCode=0
Nov 02 01:24:19.006032 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.005988    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerDied","Data":"a5d53eea0d5802e8c175bcf398b77ec81f3de2c2953fd80064066594285391ee"}
Nov 02 01:24:19.013163 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.013136    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvj9k" event={"ID":"3f612b3f-b624-45ae-b05d-2b4a164fd4af","Type":"ContainerStarted","Data":"d15f3f64be13716f6abed342b8fbc10c24b210b9cd031031f4f2ca4f52b10d75"}
Nov 02 01:24:19.230624 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.230176    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vvj9k" podStartSLOduration=1.983233667 podStartE2EDuration="6.230162641s" totalImagesPullingTime="4.246928974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:19.229193906 +0000 UTC m=+6.740217383" watchObservedRunningTime="2026-11-02 01:24:19.230162641 +0000 UTC m=+6.741186158"
Nov 02 01:24:19.381741 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.381670    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:19.382279 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.382257    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:19.942794 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.942761    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:19.942987 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:19.942889    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:19.942987 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.942968    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:19.943120 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:19.943058    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:19.943120 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:19.943110    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:19.943229 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:19.943174    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:20.018520 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:20.018479    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"8c8e4877a18ae50f2a48041b0698e786","Type":"ContainerStarted","Data":"d95c5a1630380ae15d91fa4caade01bc597bb2b6ac510daba69dfc143fd8ed7e"}
Nov 02 01:24:20.025185 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:20.025089    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"0483b9f053a944bf50e838e170a69c85","Type":"ContainerStarted","Data":"f3e65ce48f01aca8dde333d945503f44e05cbf73b7c980f768c7c26ccb0969c8"}
Nov 02 01:24:20.251569 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:20.251515    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal" podStartSLOduration=7.251498861 podStartE2EDuration="7.251498861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:20.135596234 +0000 UTC m=+7.646619711" watchObservedRunningTime="2026-11-02 01:24:20.251498861 +0000 UTC m=+7.762522339"
Nov 02 01:24:21.032516 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.032473    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9pdx5" event={"ID":"6bc310c2-330e-492f-b635-5d689f1e5203","Type":"ContainerStarted","Data":"e48c772ace3c2031e3ac8fc2b4589b10267bb7246a174ca98592db612bdad516"}
Nov 02 01:24:21.036596 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.035460    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:21.036596 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.035868    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj" event={"ID":"3464bf51-0036-4948-8bb4-43f1000d58d2","Type":"ContainerStarted","Data":"15a025b20310391f23a32dbfed4d6901e0a02ebf70950ab034afe9348e35432d"}
Nov 02 01:24:21.146947 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.146887    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9pdx5" podStartSLOduration=2.529512149 podStartE2EDuration="8.146870362s" totalImagesPullingTime="5.617358213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:21.146524762 +0000 UTC m=+8.657548240" watchObservedRunningTime="2026-11-02 01:24:21.146870362 +0000 UTC m=+8.657893839"
Nov 02 01:24:21.147136 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.147091    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal" podStartSLOduration=8.14708539 podStartE2EDuration="8.14708539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:20.251746869 +0000 UTC m=+7.762770348" watchObservedRunningTime="2026-11-02 01:24:21.14708539 +0000 UTC m=+8.658108868"
Nov 02 01:24:21.776162 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.775662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:21.776162 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.775707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:21.776162 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.775819    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:21.776162 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.775863    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:21.776162 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.775904    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs podName:55451071-26c3-4ebd-9377-d5cc1b967ea5 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:29.775891468 +0000 UTC m=+17.286914935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs") pod "network-metrics-daemon-26vc2" (UID: "55451071-26c3-4ebd-9377-d5cc1b967ea5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:21.776162 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.775927    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret podName:1955308f-bad5-47ef-951a-95b7064ed5b2 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:29.775910357 +0000 UTC m=+17.286933828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret") pod "global-pull-secret-syncer-hcbh5" (UID: "1955308f-bad5-47ef-951a-95b7064ed5b2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:21.876097 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.876065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:21.876277 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.876252    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:21.876277 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.876271    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:21.876386 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.876283    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pwtcv for pod openshift-network-diagnostics/network-check-target-wjrcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:21.876386 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.876337    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv podName:e7237264-dbfe-4566-8e42-8ddc59360817 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:29.876320155 +0000 UTC m=+17.387343624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwtcv" (UniqueName: "kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv") pod "network-check-target-wjrcp" (UID: "e7237264-dbfe-4566-8e42-8ddc59360817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:21.942233 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.942205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:21.942475 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.942360    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:21.942475 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.942376    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:21.942624 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.942491    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:21.942624 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:21.942518    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:21.942624 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:21.942598    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:22.363225 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:22.363106    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:24:22.884300 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:22.884195    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:24:22.363126098Z","UUID":"3f7eef06-7a45-47a9-9feb-13b9ca9014f0","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:24:22.885967 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:22.885944    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:24:22.886103 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:22.885981    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:24:23.044813 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:23.044782    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kbt65" event={"ID":"9766361f-a2b3-4c58-9f47-899e699c2520","Type":"ContainerStarted","Data":"b6204ab413306948188d6f9c419be5aac7fd68eac024adbf4a4a283ec87c31e1"}
Nov 02 01:24:23.048336 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:23.048307    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj" event={"ID":"3464bf51-0036-4948-8bb4-43f1000d58d2","Type":"ContainerStarted","Data":"3b462350b9a06be7e4792fb5c42165c35dbc9732d44153ded23b2991a068603f"}
Nov 02 01:24:23.050453 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:23.050431    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerStarted","Data":"c965fbd6ce6d1ca806836d2d56677cb37458593575f90a607531d849ad6f9a0f"}
Nov 02 01:24:23.164779 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:23.163997    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kbt65" podStartSLOduration=2.353341936 podStartE2EDuration="10.163982627s" totalImagesPullingTime="7.810640691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:23.163953278 +0000 UTC m=+10.674976755" watchObservedRunningTime="2026-11-02 01:24:23.163982627 +0000 UTC m=+10.675006103"
Nov 02 01:24:23.942935 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:23.942911    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:23.943238 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:23.943016    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:23.943238 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:23.943078    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:23.943238 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:23.943129    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:23.943238 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:23.943162    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:23.943238 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:23.943201    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:24.056267 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:24.056229    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sf489" event={"ID":"ba2de926-a683-49c8-9d4d-f45a09702479","Type":"ContainerStarted","Data":"1b17862aa4ad92f9f76a58b76c51d1e67265320b3379fa02675562c5be8f08d6"}
Nov 02 01:24:24.060237 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:24.059579    2574 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="c965fbd6ce6d1ca806836d2d56677cb37458593575f90a607531d849ad6f9a0f" exitCode=0
Nov 02 01:24:24.060237 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:24.059684    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerDied","Data":"c965fbd6ce6d1ca806836d2d56677cb37458593575f90a607531d849ad6f9a0f"}
Nov 02 01:24:24.064166 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:24.064138    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj" event={"ID":"3464bf51-0036-4948-8bb4-43f1000d58d2","Type":"ContainerStarted","Data":"4a7b3d3eb94307c06cba0bed7ba23eb4e964f1eeb23f9ed6755389a4102ea821"}
Nov 02 01:24:24.171148 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:24.171044    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sf489" podStartSLOduration=3.31144621 podStartE2EDuration="11.171029631s" totalImagesPullingTime="7.859583421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:24.170511477 +0000 UTC m=+11.681534955" watchObservedRunningTime="2026-11-02 01:24:24.171029631 +0000 UTC m=+11.682053106"
Nov 02 01:24:24.429250 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:24.429157    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj" podStartSLOduration=2.385958204 podStartE2EDuration="11.429139321s" totalImagesPullingTime="9.043181117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:24.428758867 +0000 UTC m=+11.939782344" watchObservedRunningTime="2026-11-02 01:24:24.429139321 +0000 UTC m=+11.940162798"
Nov 02 01:24:25.068003 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:25.067772    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7pbwr" event={"ID":"7d20a7d0-d504-4b4d-89a6-2c48b5de6315","Type":"ContainerStarted","Data":"ef34d0da83891248527fb0f19da3a21cedbf80ad429265799fc8db7c85d331ce"}
Nov 02 01:24:25.942512 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:25.942483    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:25.942512 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:25.942487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:25.942772 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:25.942686    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:25.942908 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:25.942883    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:25.943001 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:25.942977    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:25.943076 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:25.943060    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:26.058587 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:26.058541    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7pbwr" podStartSLOduration=3.2411659999999998 podStartE2EDuration="13.058521774s" totalImagesPullingTime="9.817355774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:25.187752409 +0000 UTC m=+12.698775889" watchObservedRunningTime="2026-11-02 01:24:26.058521774 +0000 UTC m=+13.569545252"
Nov 02 01:24:26.071981 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:26.071957    2574 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="828668b816eb278f57a742165f3e58f24575b831839585842597f9a859d9f569" exitCode=0
Nov 02 01:24:26.072483 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:26.072061    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerDied","Data":"828668b816eb278f57a742165f3e58f24575b831839585842597f9a859d9f569"}
Nov 02 01:24:27.075720 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:27.075679    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q77dv" event={"ID":"87856a71-2d8f-47f8-8d85-4d797fb8bec4","Type":"ContainerStarted","Data":"fcf70cd32be14e796944c3a7e8692d22c07a8ee0431fbb7db4f03349e48f6791"}
Nov 02 01:24:27.942714 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:27.942669    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:27.942860 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:27.942811    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:27.942860 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:27.942680    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:27.942942 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:27.942897    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:27.942942 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:27.942669    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:27.943009 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:27.942948    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:28.081385 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:28.081348    2574 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="b3dbe5e9b4b6c5a01f2e1a1d03ad7187debaac708bf11b9d4ce55def5fee35c4" exitCode=0
Nov 02 01:24:28.081858 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:28.081416    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerDied","Data":"b3dbe5e9b4b6c5a01f2e1a1d03ad7187debaac708bf11b9d4ce55def5fee35c4"}
Nov 02 01:24:28.234447 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:28.234053    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q77dv" podStartSLOduration=3.82767523 podStartE2EDuration="15.234041241s" totalImagesPullingTime="11.406366011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:27.190075574 +0000 UTC m=+14.701099051" watchObservedRunningTime="2026-11-02 01:24:28.234041241 +0000 UTC m=+15.745064716"
Nov 02 01:24:29.086492 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.086470    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:24:29.086916 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.086892    2574 generic.go:356] "Generic (PLEG): container finished" podID="0406fb90-83ff-40c9-a3aa-e834abf84e46" containerID="d0b3718de4cdc63735591fcbf1cbce933c314f6153c2330e42232933d82e7275" exitCode=1
Nov 02 01:24:29.086988 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.086940    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"aaaae9f6d22e01de2eefe994f94042f29958d5495656453a3f7fc9aa2a4e0279"}
Nov 02 01:24:29.087040 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.087014    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerDied","Data":"d0b3718de4cdc63735591fcbf1cbce933c314f6153c2330e42232933d82e7275"}
Nov 02 01:24:29.087088 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.087063    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"eaa8b4a4a0a14fafcedd675a37bba7a3e36b852912e5f390c5f7e564a30a176b"}
Nov 02 01:24:29.842772 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.842524    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:29.842940 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.842829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:29.842940 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.842704    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:29.842940 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.842928    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs podName:55451071-26c3-4ebd-9377-d5cc1b967ea5 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.842903619 +0000 UTC m=+33.353927100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs") pod "network-metrics-daemon-26vc2" (UID: "55451071-26c3-4ebd-9377-d5cc1b967ea5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 01:24:29.842940 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.842937    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:29.843120 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.842981    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret podName:1955308f-bad5-47ef-951a-95b7064ed5b2 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.842970882 +0000 UTC m=+33.353994373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret") pod "global-pull-secret-syncer-hcbh5" (UID: "1955308f-bad5-47ef-951a-95b7064ed5b2") : object "kube-system"/"original-pull-secret" not registered
Nov 02 01:24:29.942937 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.942906    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:29.943119 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.943039    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:29.943119 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.942921    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:29.943119 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.943084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:29.943275 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.943148    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:29.943275 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.943211    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:29.943275 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:29.943217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:29.943417 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.943329    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 01:24:29.943417 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.943342    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 01:24:29.943417 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.943349    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pwtcv for pod openshift-network-diagnostics/network-check-target-wjrcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:29.943417 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:29.943389    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv podName:e7237264-dbfe-4566-8e42-8ddc59360817 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.943373707 +0000 UTC m=+33.454397175 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwtcv" (UniqueName: "kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv") pod "network-check-target-wjrcp" (UID: "e7237264-dbfe-4566-8e42-8ddc59360817") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 01:24:30.092264 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:30.092246    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:24:30.092651 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:30.092627    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"2b63cf31911d8abfda2a9b15af7af26e1ab92a5505c3b099e85070df155f68af"}
Nov 02 01:24:30.092741 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:30.092714    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"804c2418bfad69b3654659e5ade18b7957b6a0e750bbf1c07c3ce977ae89017f"}
Nov 02 01:24:30.092798 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:30.092770    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"b0aa472378707dff526dd52b4305c151eba7ae931c77b2c5ab6a248606a22c97"}
Nov 02 01:24:30.095413 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:30.095393    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerStarted","Data":"16515c20a5258e186d54f0f8d328d3075e046bb5fa1c4063b096c2fcc5376516"}
Nov 02 01:24:31.099518 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:31.099484    2574 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="16515c20a5258e186d54f0f8d328d3075e046bb5fa1c4063b096c2fcc5376516" exitCode=0
Nov 02 01:24:31.099973 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:31.099531    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerDied","Data":"16515c20a5258e186d54f0f8d328d3075e046bb5fa1c4063b096c2fcc5376516"}
Nov 02 01:24:31.942362 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:31.942327    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:31.942362 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:31.942359    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:31.942362 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:31.942355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:31.942586 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:31.942431    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:31.942586 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:31.942483    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:31.942586 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:31.942550    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:32.104722 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:32.104692    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:24:32.105183 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:32.105064    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"ceeecc32276438e690005dbd18e67e8a4717051c98a074f92012305fc7cab296"}
Nov 02 01:24:32.107992 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:32.107964    2574 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="9b842414db61277b1ed031fd7c12b46c218f41f5631c108fa87d08f924d42cb9" exitCode=0
Nov 02 01:24:32.108094 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:32.108007    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerDied","Data":"9b842414db61277b1ed031fd7c12b46c218f41f5631c108fa87d08f924d42cb9"}
Nov 02 01:24:33.113555 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:33.113520    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" event={"ID":"46964574-0965-4bf3-8ce7-1c3c5115c594","Type":"ContainerStarted","Data":"d4ea391e0051f0a3d1b3620f9d2d999f0df08ce449eac74f3fff7a8215997777"}
Nov 02 01:24:33.335499 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:33.335450    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cdbt2" podStartSLOduration=9.070030033 podStartE2EDuration="20.335437494s" totalImagesPullingTime="11.265407461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:33.334870348 +0000 UTC m=+20.845893823" watchObservedRunningTime="2026-11-02 01:24:33.335437494 +0000 UTC m=+20.846460969"
Nov 02 01:24:33.943031 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:33.942995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:33.943031 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:33.943033    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:33.943218 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:33.943109    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:33.943218 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:33.943115    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:33.943218 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:33.943183    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:33.943311 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:33.943258    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:34.119313 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:34.119282    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:24:34.119997 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:34.119719    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"56401efc225f9b86a30a861bc876cb16ac89da319d4d6a0482ee84cda6c7acc2"}
Nov 02 01:24:34.120313 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:34.120293    2574 scope.go:122] "RemoveContainer" containerID="d0b3718de4cdc63735591fcbf1cbce933c314f6153c2330e42232933d82e7275"
Nov 02 01:24:35.125004 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.124976    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:24:35.125408 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.125318    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" event={"ID":"0406fb90-83ff-40c9-a3aa-e834abf84e46","Type":"ContainerStarted","Data":"e0c1a7f7b089da793ba6bdfdba0fcf4fc992b2507455bcac3900a3f0d3c685bf"}
Nov 02 01:24:35.125449 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.125414    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:35.125871 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.125835    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:35.126013 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.125896    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:35.140962 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.140941    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:35.141068 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.141030    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:35.403109 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.403012    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn" podStartSLOduration=8.051382 podStartE2EDuration="22.403000225s" totalImagesPullingTime="14.351618225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:35.402917913 +0000 UTC m=+22.913941401" watchObservedRunningTime="2026-11-02 01:24:35.403000225 +0000 UTC m=+22.914023701"
Nov 02 01:24:35.942798 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.942768    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:35.942971 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.942843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:35.942971 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:35.942933    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:35.942971 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:35.942949    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:35.943134 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:35.943058    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:35.943184 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:35.943149    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:36.127280 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:36.127255    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:36.158934 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:36.158908    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hcbh5"]
Nov 02 01:24:36.159055 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:36.158992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:36.159101 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:36.159066    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:36.162953 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:36.162925    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wjrcp"]
Nov 02 01:24:36.163075 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:36.163014    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:36.163135 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:36.163095    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:36.163756 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:36.163732    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-26vc2"]
Nov 02 01:24:36.163847 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:36.163830    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:36.163954 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:36.163934    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:37.129195 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:37.129171    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:37.942922 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:37.942890    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:37.942922 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:37.942917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:37.943137 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:37.942936    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:37.943137 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:37.943031    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:37.943236 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:37.943135    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:37.943236 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:37.943202    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:39.942191 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:39.942092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:39.942191 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:39.942092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:39.942654 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:39.942199    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wjrcp" podUID="e7237264-dbfe-4566-8e42-8ddc59360817"
Nov 02 01:24:39.942654 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:39.942092    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:39.942654 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:39.942283    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hcbh5" podUID="1955308f-bad5-47ef-951a-95b7064ed5b2"
Nov 02 01:24:39.942654 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:39.942406    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-26vc2" podUID="55451071-26c3-4ebd-9377-d5cc1b967ea5"
Nov 02 01:24:41.346638 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:41.346540    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeReady"
Nov 02 01:24:41.347051 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:41.346673    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 01:24:41.943048 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:41.942952    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:41.943224 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:41.943063    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:41.943224 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:41.943104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:41.997870 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:41.997838    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:24:41.997998 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:41.997875    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:24:41.998034 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:41.998011    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:24:42.047640 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:42.047595    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qtccq\""
Nov 02 01:24:42.047792 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:42.047683    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6qqtd\""
Nov 02 01:24:42.047836 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:42.047821    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:24:42.047916 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:42.047902    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82k56\""
Nov 02 01:24:42.464577 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:42.464540    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:42.465023 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:42.464733    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:42.465218 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:42.465203    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:43.402959 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:43.402925    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:43.403167 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:43.403153    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:24:43.422528 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:43.422503    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:44.286422 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:44.286394    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q77dv_87856a71-2d8f-47f8-8d85-4d797fb8bec4/dns-node-resolver/0.log"
Nov 02 01:24:44.829069 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:44.829044    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9pdx5_6bc310c2-330e-492f-b635-5d689f1e5203/node-ca/0.log"
Nov 02 01:24:45.854346 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:45.854311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:45.854776 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:45.854368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:45.854776 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:45.854444    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 01:24:45.854776 ip-10-0-134-183 kubenswrapper[2574]: E1102 01:24:45.854512    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs podName:55451071-26c3-4ebd-9377-d5cc1b967ea5 nodeName:}" failed. No retries permitted until 2026-11-02 01:25:17.854496032 +0000 UTC m=+65.365519486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs") pod "network-metrics-daemon-26vc2" (UID: "55451071-26c3-4ebd-9377-d5cc1b967ea5") : secret "metrics-daemon-secret" not found
Nov 02 01:24:45.858040 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:45.858024    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:45.859969 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:45.859955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:45.955647 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:45.955595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:45.958117 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:45.958098    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:46.171773 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:46.171693    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:46.399323 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:46.399293    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hcbh5"]
Nov 02 01:24:46.402677 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:46.402653    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1955308f_bad5_47ef_951a_95b7064ed5b2.slice/crio-132e89e8ef04df509abb9d5b00424e21106ab48891aa52a42b124f23d58e2a74 WatchSource:0}: Error finding container 132e89e8ef04df509abb9d5b00424e21106ab48891aa52a42b124f23d58e2a74: Status 404 returned error can't find the container with id 132e89e8ef04df509abb9d5b00424e21106ab48891aa52a42b124f23d58e2a74
Nov 02 01:24:46.538849 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:46.538821    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wjrcp"]
Nov 02 01:24:46.542390 ip-10-0-134-183 kubenswrapper[2574]: W1102 01:24:46.542366    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7237264_dbfe_4566_8e42_8ddc59360817.slice/crio-03d8e0ab017b9164b69426880d82ee1f8b2e0508e63a12ed3be0bea3f06ab930 WatchSource:0}: Error finding container 03d8e0ab017b9164b69426880d82ee1f8b2e0508e63a12ed3be0bea3f06ab930: Status 404 returned error can't find the container with id 03d8e0ab017b9164b69426880d82ee1f8b2e0508e63a12ed3be0bea3f06ab930
Nov 02 01:24:47.151480 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:47.151443    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wjrcp" event={"ID":"e7237264-dbfe-4566-8e42-8ddc59360817","Type":"ContainerStarted","Data":"03d8e0ab017b9164b69426880d82ee1f8b2e0508e63a12ed3be0bea3f06ab930"}
Nov 02 01:24:47.152484 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:47.152464    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hcbh5" event={"ID":"1955308f-bad5-47ef-951a-95b7064ed5b2","Type":"ContainerStarted","Data":"132e89e8ef04df509abb9d5b00424e21106ab48891aa52a42b124f23d58e2a74"}
Nov 02 01:24:51.164779 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:51.164738    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wjrcp" event={"ID":"e7237264-dbfe-4566-8e42-8ddc59360817","Type":"ContainerStarted","Data":"e06e44c3e7525c089b57ff5079de96f8e16378a72fc88edcb4b23cbdfc73f15e"}
Nov 02 01:24:51.165216 ip-10-0-134-183 kubenswrapper[2574]: I1102 01:24:51.164888    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:52.029810 ip-10-0-134-183 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 01:24:52.044184 ip-10-0-134-183 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 01:24:52.044352 ip-10-0-134-183 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 01:24:52.044398 ip-10-0-134-183 systemd[1]: kubelet.service: Consumed 2.129s CPU time.
Nov 02 01:24:52.054215 ip-10-0-134-183 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 01:24:52.155300 ip-10-0-134-183 kubenswrapper[4287]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:52.155300 ip-10-0-134-183 kubenswrapper[4287]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 01:24:52.155300 ip-10-0-134-183 kubenswrapper[4287]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:52.155300 ip-10-0-134-183 kubenswrapper[4287]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 01:24:52.157413 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157330    4287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:52.157413 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157413    4287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:52.157413 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157417    4287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157421    4287 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157425    4287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157428    4287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157431    4287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157434    4287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157444    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157446    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157449    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157452    4287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157455    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157457    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157460    4287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157462    4287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157465    4287 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157468    4287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157470    4287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157473    4287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157475    4287 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157478    4287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:52.157515 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157481    4287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157483    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157486    4287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157489    4287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157492    4287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157494    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157498    4287 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157501    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157503    4287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157505    4287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157508    4287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157510    4287 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157513    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157516    4287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157519    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157522    4287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157524    4287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157528    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157531    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157534    4287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:52.158038 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157536    4287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157539    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157541    4287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157544    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157547    4287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157550    4287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157553    4287 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157555    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157559    4287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157562    4287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157564    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157567    4287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157570    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157573    4287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157576    4287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157578    4287 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157581    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157583    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157587    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157590    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:52.158539 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157593    4287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157595    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157598    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157601    4287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157620    4287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157624    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157628    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157632    4287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157636    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157640    4287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157643    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157646    4287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157649    4287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157651    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157654    4287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157656    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157660    4287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157662    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157665    4287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:52.159043 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157669    4287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157674    4287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157678    4287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157681    4287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157685    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157687    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157690    4287 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157693    4287 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157698    4287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157702    4287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157705    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157707    4287 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157711    4287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157713    4287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157716    4287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157719    4287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157721    4287 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157724    4287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157726    4287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:52.159500 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157729    4287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157731    4287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157734    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157736    4287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157739    4287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.157742    4287 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158507    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158514    4287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158518    4287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158520    4287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158523    4287 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158526    4287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158529    4287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158532    4287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158534    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158537    4287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158540    4287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158546    4287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158550    4287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:52.159977 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158554    4287 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158557    4287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158559    4287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158562    4287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158566    4287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158568    4287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158571    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158574    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158576    4287 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158579    4287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158582    4287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158585    4287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158587    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158590    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158593    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158595    4287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158598    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158600    4287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158620    4287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:52.160458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158625    4287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158629    4287 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158633    4287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158637    4287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158640    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158643    4287 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158646    4287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158648    4287 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158651    4287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158653    4287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158656    4287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158659    4287 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158663    4287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158667    4287 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158670    4287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158673    4287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158676    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158678    4287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158681    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158684    4287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:52.160936 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158686    4287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158689    4287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158691    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158694    4287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158696    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158699    4287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158701    4287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158704    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158706    4287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158709    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158711    4287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158714    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158716    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158720    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158722    4287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158726    4287 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158728    4287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158731    4287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158734    4287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:52.161475 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158736    4287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158739    4287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158741    4287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158744    4287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158746    4287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158748    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158751    4287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158753    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158756    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158759    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158762    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158764    4287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158767    4287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158769    4287 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158772    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158774    4287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158777    4287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158779    4287 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158781    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158784    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:52.161962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158787    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158789    4287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158793    4287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158795    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158798    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158801    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158803    4287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158806    4287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158809    4287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158812    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158814    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158817    4287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158819    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158822    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.158824    4287 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159301    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159306    4287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159309    4287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159312    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159315    4287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:52.162467 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159318    4287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159320    4287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159325    4287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159328    4287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159331    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159334    4287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159337    4287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159339    4287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159342    4287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159345    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159347    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159350    4287 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159353    4287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159356    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159358    4287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159361    4287 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159364    4287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159366    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159368    4287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:52.162987 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159373    4287 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159376    4287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159379    4287 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159381    4287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159384    4287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159386    4287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159389    4287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159391    4287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159394    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159397    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159399    4287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159402    4287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159404    4287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159406    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159409    4287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159413    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159415    4287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159418    4287 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159420    4287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159422    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:52.163453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159425    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159428    4287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159430    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159433    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159436    4287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159438    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159440    4287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159443    4287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159445    4287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159448    4287 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159450    4287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159453    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159455    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159458    4287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159461    4287 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159463    4287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159466    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159468    4287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159470    4287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:52.163962 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159473    4287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159475    4287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159478    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159480    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159483    4287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159485    4287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159488    4287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159491    4287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159494    4287 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159496    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159499    4287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159501    4287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159504    4287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159507    4287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159510    4287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159512    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159515    4287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159517    4287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159520    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:52.164432 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159522    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159525    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159527    4287 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159529    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159533    4287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159537    4287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159540    4287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159543    4287 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159546    4287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159548    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159551    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159553    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159556    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159558    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159561    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159564    4287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159566    4287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159569    4287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159571    4287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159577    4287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:52.164897 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159579    4287 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159582    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159584    4287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.159587    4287 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159695    4287 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159704    4287 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159715    4287 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159723    4287 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159728    4287 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159732    4287 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159736    4287 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159740    4287 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159743    4287 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159746    4287 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159750    4287 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159753    4287 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159756    4287 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159759    4287 flags.go:64] FLAG: --cgroup-root=""
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159761    4287 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159764    4287 flags.go:64] FLAG: --client-ca-file=""
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159767    4287 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159772    4287 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159776    4287 flags.go:64] FLAG: --cluster-domain=""
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159779    4287 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 01:24:52.165376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159782    4287 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159785    4287 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159788    4287 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159792    4287 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159795    4287 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159798    4287 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159802    4287 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159808    4287 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159811    4287 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159814    4287 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159817    4287 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159820    4287 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159824    4287 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159827    4287 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159830    4287 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159833    4287 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159836    4287 flags.go:64] FLAG: --enable-server="true"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159839    4287 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159847    4287 flags.go:64] FLAG: --event-burst="100"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159850    4287 flags.go:64] FLAG: --event-qps="50"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159852    4287 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159855    4287 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159858    4287 flags.go:64] FLAG: --eviction-hard=""
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159863    4287 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 01:24:52.166237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159865    4287 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159869    4287 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159872    4287 flags.go:64] FLAG: --eviction-soft=""
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159875    4287 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159878    4287 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159881    4287 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159884    4287 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159887    4287 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159890    4287 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159893    4287 flags.go:64] FLAG: --feature-gates=""
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159899    4287 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159902    4287 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159906    4287 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159909    4287 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159912    4287 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159917    4287 flags.go:64] FLAG: --help="false"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159920    4287 flags.go:64] FLAG: --hostname-override="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159923    4287 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159927    4287 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159930    4287 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159933    4287 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159936    4287 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159940    4287 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 01:24:52.167049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159942    4287 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159945    4287 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159948    4287 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159951    4287 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159954    4287 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159957    4287 flags.go:64] FLAG: --kube-reserved=""
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159960    4287 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159963    4287 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159966    4287 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159969    4287 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159971    4287 flags.go:64] FLAG: --lock-file=""
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159974    4287 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159977    4287 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159980    4287 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159987    4287 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159989    4287 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159993    4287 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159996    4287 flags.go:64] FLAG: --logging-format="text"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.159999    4287 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160002    4287 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160005    4287 flags.go:64] FLAG: --manifest-url=""
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160008    4287 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160013    4287 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160016    4287 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160021    4287 flags.go:64] FLAG: --max-pods="110"
Nov 02 01:24:52.167627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160024    4287 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160027    4287 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160030    4287 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160033    4287 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160036    4287 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160039    4287 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160042    4287 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160049    4287 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160052    4287 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160055    4287 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160059    4287 flags.go:64] FLAG: --pod-cidr=""
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160061    4287 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160064    4287 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160067    4287 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160070    4287 flags.go:64] FLAG: --port="10250"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160073    4287 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160076    4287 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01e69977d1c5fe041"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160079    4287 flags.go:64] FLAG: --qos-reserved=""
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160082    4287 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160085    4287 flags.go:64] FLAG: --register-node="true"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160088    4287 flags.go:64] FLAG: --register-with-taints=""
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160092    4287 flags.go:64] FLAG: --registry-burst="10"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160095    4287 flags.go:64] FLAG: --registry-qps="5"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160097    4287 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160101    4287 flags.go:64] FLAG: --reserved-memory=""
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160105    4287 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 01:24:52.168249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160108    4287 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160111    4287 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160114    4287 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160117    4287 flags.go:64] FLAG: --runonce="false"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160120    4287 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160125    4287 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160128    4287 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160130    4287 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160133    4287 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160136    4287 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160141    4287 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160144    4287 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160147    4287 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160150    4287 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160153    4287 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160155    4287 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160158    4287 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160161    4287 flags.go:64] FLAG: --system-cgroups=""
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160164    4287 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160169    4287 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160172    4287 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160175    4287 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160179    4287 flags.go:64] FLAG: --tls-min-version=""
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160182    4287 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160185    4287 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 01:24:52.168875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160188    4287 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160191    4287 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160194    4287 flags.go:64] FLAG: --v="2"
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160198    4287 flags.go:64] FLAG: --version="false"
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160202    4287 flags.go:64] FLAG: --vmodule=""
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160206    4287 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160209    4287 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160356    4287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160360    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160363    4287 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160366    4287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160369    4287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160373    4287 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160376    4287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160379    4287 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160381    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160384    4287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160388    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160390    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160393    4287 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160395    4287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160398    4287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:52.169458 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160400    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160403    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160405    4287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160408    4287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160411    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160413    4287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160416    4287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160418    4287 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160421    4287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160423    4287 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160426    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160428    4287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160431    4287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160433    4287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160436    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160438    4287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160441    4287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160444    4287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160446    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160449    4287 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:52.170111 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160451    4287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160454    4287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160457    4287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160460    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160463    4287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160466    4287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160468    4287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160473    4287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160475    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160478    4287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160481    4287 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160483    4287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160485    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160488    4287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160491    4287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160493    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160497    4287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160501    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160504    4287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:52.170594 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160506    4287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160509    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160511    4287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160514    4287 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160516    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160519    4287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160521    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160524    4287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160526    4287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160529    4287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160531    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160534    4287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160537    4287 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160540    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160543    4287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160546    4287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160549    4287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160552    4287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160555    4287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:52.171093 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160558    4287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160561    4287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160564    4287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160566    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160569    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160571    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160574    4287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160576    4287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160579    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160581    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160584    4287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160586    4287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160589    4287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160591    4287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160594    4287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160596    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160599    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160601    4287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160623    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160627    4287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:52.171600 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160630    4287 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160632    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160635    4287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160637    4287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160640    4287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160644    4287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160648    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160651    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160655    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160658    4287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160661    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160663    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160666    4287 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.160695    4287 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160731    4287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:52.172207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160737    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160740    4287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160744    4287 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160747    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160749    4287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160752    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160755    4287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160757    4287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160760    4287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160762    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160765    4287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160768    4287 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160770    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160773    4287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160775    4287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160778    4287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160780    4287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160783    4287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160786    4287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160788    4287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:52.172576 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160791    4287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160793    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160796    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160799    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160802    4287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160805    4287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160808    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160811    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160814    4287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160816    4287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160820    4287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160822    4287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160825    4287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160827    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160830    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160832    4287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160835    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160838    4287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160840    4287 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160843    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:52.173272 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160845    4287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160847    4287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160850    4287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160854    4287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160857    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160860    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160863    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160865    4287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160868    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160870    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160873    4287 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160875    4287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160878    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160880    4287 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160883    4287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160885    4287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160888    4287 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160891    4287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160894    4287 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160897    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:52.173848 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160899    4287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160902    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160905    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160908    4287 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160911    4287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160913    4287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160916    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160918    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160920    4287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160923    4287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160926    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160928    4287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160931    4287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160933    4287 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160936    4287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160939    4287 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160942    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160945    4287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160948    4287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160950    4287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:52.174339 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160953    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160955    4287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160958    4287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160960    4287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160963    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160965    4287 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160968    4287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160970    4287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160973    4287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160976    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160979    4287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160981    4287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160984    4287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160987    4287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160990    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160993    4287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160995    4287 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.160998    4287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.161000    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:52.174875 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.161003    4287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.161005    4287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.161008    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.161013    4287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.161016    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.161019    4287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.164882    4287 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.164899    4287 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.164919    4287 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.164923    4287 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.164979    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.164984    4287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.164988    4287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.164991    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.164993    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.164996    4287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.164999    4287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165001    4287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:52.175322 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165004    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165006    4287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165009    4287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165012    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165015    4287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165018    4287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165020    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165023    4287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165026    4287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165029    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165032    4287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165034    4287 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165037    4287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165040    4287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165042    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165045    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165048    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165050    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165053    4287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:52.175774 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165055    4287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165058    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165062    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165066    4287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165070    4287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165073    4287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165076    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165079    4287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165082    4287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165085    4287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165087    4287 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165090    4287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165093    4287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165095    4287 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165098    4287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165102    4287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165105    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165108    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:52.176225 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165111    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165114    4287 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165116    4287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165119    4287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165121    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165123    4287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165126    4287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165128    4287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165131    4287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165133    4287 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165136    4287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165138    4287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165141    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165144    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165146    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165149    4287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165152    4287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165155    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165158    4287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165161    4287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:52.176673 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165163    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165165    4287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165168    4287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165171    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165173    4287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165176    4287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165178    4287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165181    4287 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165183    4287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165186    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165188    4287 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165191    4287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165193    4287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165196    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165198    4287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165201    4287 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165203    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165206    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165208    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165211    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:52.177152 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165214    4287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165216    4287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165219    4287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165221    4287 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165224    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165226    4287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165229    4287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165231    4287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165234    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165237    4287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165242    4287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165245    4287 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165248    4287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165250    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165253    4287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165256    4287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165258    4287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165261    4287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165264    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165266    4287 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:52.177703 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165269    4287 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.165296    4287 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165435    4287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165440    4287 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165443    4287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165445    4287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165448    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165450    4287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165453    4287 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165456    4287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165458    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165461    4287 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165464    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165466    4287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165469    4287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165471    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:52.178175 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165474    4287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165476    4287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165479    4287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165481    4287 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165484    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165486    4287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165489    4287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165492    4287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165494    4287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165497    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165499    4287 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165502    4287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165504    4287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165507    4287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165509    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165511    4287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165514    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165516    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165519    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:52.178558 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165521    4287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165523    4287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165526    4287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165529    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165531    4287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165534    4287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165536    4287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165539    4287 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165541    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165543    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165546    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165548    4287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165551    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165554    4287 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165556    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165558    4287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165561    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165563    4287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165566    4287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165568    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:52.179026 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165570    4287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165621    4287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.165659    4287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166012    4287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166019    4287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166023    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166028    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166033    4287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166037    4287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166043    4287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166048    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166052    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166057    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166062    4287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166066    4287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166076    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166081    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166085    4287 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166089    4287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:52.179509 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166094    4287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166098    4287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166102    4287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166106    4287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166111    4287 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166116    4287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166121    4287 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166129    4287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166141    4287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166147    4287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166151    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166155    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166160    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166164    4287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166168    4287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166174    4287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166178    4287 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166183    4287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166187    4287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166190    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:52.179959 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166195    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166204    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166208    4287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166212    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166216    4287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166220    4287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166225    4287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166238    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166243    4287 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166249    4287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166257    4287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166262    4287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166267    4287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166277    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.166335    4287 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166498    4287 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 01:24:52.180460 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166507    4287 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166512    4287 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166521    4287 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166526    4287 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166530    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166534    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166538    4287 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166542    4287 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166553    4287 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166558    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166562    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166566    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166572    4287 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166577    4287 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166581    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166585    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166590    4287 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166594    4287 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166598    4287 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 01:24:52.180854 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166602    4287 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166629    4287 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166638    4287 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166642    4287 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166647    4287 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166651    4287 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166658    4287 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166663    4287 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166670    4287 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166675    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166679    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166683    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166687    4287 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166697    4287 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166702    4287 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166707    4287 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166712    4287 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166716    4287 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166720    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 01:24:52.181307 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166724    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166728    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166732    4287 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166736    4287 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166745    4287 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166748    4287 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166751    4287 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166755    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166758    4287 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166762    4287 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166767    4287 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166770    4287 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166773    4287 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166776    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166779    4287 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166782    4287 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166790    4287 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166927    4287 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166938    4287 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 01:24:52.181771 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166942    4287 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166945    4287 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166950    4287 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166953    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166956    4287 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166958    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166961    4287 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166964    4287 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166966    4287 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166971    4287 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166974    4287 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166976    4287 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166979    4287 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166981    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166984    4287 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166986    4287 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166989    4287 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166992    4287 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166994    4287 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166996    4287 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 01:24:52.182404 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.166999    4287 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167002    4287 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167005    4287 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167007    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167010    4287 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167013    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167015    4287 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167017    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167020    4287 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167022    4287 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167025    4287 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167027    4287 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167030    4287 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167032    4287 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167037    4287 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167039    4287 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167042    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167044    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167047    4287 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 01:24:52.183248 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167049    4287 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167053    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167059    4287 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167062    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167065    4287 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167068    4287 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167070    4287 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167073    4287 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:52.167075    4287 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.167138    4287 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.168137    4287 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.168190    4287 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.168358    4287 server.go:1014] "Starting client certificate rotation"
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.168505    4287 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.168569    4287 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:19:12 +0000 UTC" deadline="2028-07-13 07:11:10.294657336 +0000 UTC"
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.168624    4287 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14861h46m18.126055866s"
Nov 02 01:24:52.184117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.168888    4287 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:24:52.184584 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.169956    4287 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 01:24:52.184584 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.171391    4287 log.go:25] "Validated CRI v1 runtime API"
Nov 02 01:24:52.184584 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.172916    4287 log.go:25] "Validated CRI v1 image API"
Nov 02 01:24:52.184584 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.173777    4287 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 01:24:52.184584 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.175164    4287 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 95cbff6b-6ab0-43bf-8329-a3b10046fe6e:/dev/nvme0n1p4 b70a65a0-5ff9-41c8-bcac-70b5661b8df4:/dev/nvme0n1p3]
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.175184    4287 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0048a8958faf75002bcbc53d3b5affbd0bb3cdece1237372bc495e39b454dd8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0048a8958faf75002bcbc53d3b5affbd0bb3cdece1237372bc495e39b454dd8e/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/029b8e4b89efc967f55fb182cfc26eb92d8e669fa4d0685c46ccd65b850d1543/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/029b8e4b89efc967f55fb182cfc26eb92d8e669fa4d0685c46ccd65b850d1543/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03d8e0ab017b9164b69426880d82ee1f8b2e0508e63a12ed3be0bea3f06ab930/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03d8e0ab017b9164b69426880d82ee1f8b2e0508e63a12ed3be0bea3f06ab930/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b0586fd638a3ee62d41a31d7810c079d0a958dd8976fd2fda174951f4399d34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b0586fd638a3ee62d41a31d7810c079d0a958dd8976fd2fda174951f4399d34/userdata/shm major:0 minor:65 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/132e89e8ef04df509abb9d5b00424e21106ab48891aa52a42b124f23d58e2a74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/132e89e8ef04df509abb9d5b00424e21106ab48891aa52a42b124f23d58e2a74/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1501fcf156f5fa7938f47025ea1a96be8045f2b27bd3cb56979230a4be6f035c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1501fcf156f5fa7938f47025ea1a96be8045f2b27bd3cb56979230a4be6f035c/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9348b2b7de5c8002e9acccaa25ccd54b4a9c0aa73e50d889c3d5e39f1726b304/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9348b2b7de5c8002e9acccaa25ccd54b4a9c0aa73e50d889c3d5e39f1726b304/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0fdcd2d101d0950f75b48340425da91c9b4b6c529ab59b8f8e4f0540948f95b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0fdcd2d101d0950f75b48340425da91c9b4b6c529ab59b8f8e4f0540948f95b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b121f4d54e4477af976f7a35e0fbad8412c0f6208dada04f44275e5fda0c9161/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b121f4d54e4477af976f7a35e0fbad8412c0f6208dada04f44275e5fda0c9161/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3d712a65735714262878f625e57f91c8b7a1061b2e48d852a96b2b9f974cdba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3d712a65735714262878f625e57f91c8b7a1061b2e48d852a96b2b9f974cdba/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6431d4e9b3a09c4ae90e1b5e9fcad8a814c8c87103c0094892e5c955729f9b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6431d4e9b3a09c4ae90e1b5e9fcad8a814c8c87103c0094892e5c955729f9b9/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7068729c8fb9d6c6b2e7d33c9b580b585d98a289cb68f83ed916dcc5452867c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7068729c8fb9d6c6b2e7d33c9b580b585d98a289cb68f83ed916dcc5452867c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volumes/kubernetes.io~projected/kube-api-access-77f27:{mountpoint:/var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volumes/kubernetes.io~projected/kube-api-access-77f27 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1955308f-bad5-47ef-951a-95b7064ed5b2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1955308f-bad5-47ef-951a-95b7064ed5b2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3464bf51-0036-4948-8bb4-43f1000d58d2/volumes/kubernetes.io~projected/kube-api-access-l7chh:{mountpoint:/var/lib/kubelet/pods/3464bf51-0036-4948-8bb4-43f1000d58d2/volumes/kubernetes.io~projected/kube-api-access-l7chh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f612b3f-b624-45ae-b05d-2b4a164fd4af/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3f612b3f-b624-45ae-b05d-2b4a164fd4af/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46964574-0965-4bf3-8ce7-1c3c5115c594/volumes/kubernetes.io~projected/kube-api-access-pwhmx:{mountpoint:/var/lib/kubelet/pods/46964574-0965-4bf3-8ce7-1c3c5115c594/volumes/kubernetes.io~projected/kube-api-access-pwhmx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55451071-26c3-4ebd-9377-d5cc1b967ea5/volumes/kubernetes.io~projected/kube-api-access-c2j7d:{mountpoint:/var/lib/kubelet/pods/55451071-26c3-4ebd-9377-d5cc1b967ea5/volumes/kubernetes.io~projected/kube-api-access-c2j7d major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bc310c2-330e-492f-b635-5d689f1e5203/volumes/kubernetes.io~projected/kube-api-access-vjqms:{mountpoint:/var/lib/kubelet/pods/6bc310c2-330e-492f-b635-5d689f1e5203/volumes/kubernetes.io~projected/kube-api-access-vjqms major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~projected/kube-api-access-tvvr8:{mountpoint:/var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~projected/kube-api-access-tvvr8 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87856a71-2d8f-47f8-8d85-4d797fb8bec4/volumes/kubernetes.io~projected/kube-api-access-txddj:{mountpoint:/var/lib/kubelet/pods/87856a71-2d8f-47f8-8d85-4d797fb8bec4/volumes/kubernetes.io~projected/kube-api-access-txddj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9766361f-a2b3-4c58-9f47-899e699c2520/volumes/kubernetes.io~projected/kube-api-access-bvq2v:{mountpoint:/var/lib/kubelet/pods/9766361f-a2b3-4c58-9f47-899e699c2520/volumes/kubernetes.io~projected/kube-api-access-bvq2v major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2de926-a683-49c8-9d4d-f45a09702479/volumes/kubernetes.io~projected/kube-api-access-pxns9:{mountpoint:/var/lib/kubelet/pods/ba2de926-a683-49c8-9d4d-f45a09702479/volumes/kubernetes.io~projected/kube-api-access-pxns9 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7237264-dbfe-4566-8e42-8ddc59360817/volumes/kubernetes.io~projected/kube-api-access-pwtcv:{mountpoint:/var/lib/kubelet/pods/e7237264-dbfe-4566-8e42-8ddc59360817/volumes/kubernetes.io~projected/kube-api-access-pwtcv major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f647369fb87cc9392bc99c6d1f789c9abe56c0014032d0ab74183c71d9477fe5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/eff713e61fbcb5fdbc257f9b17794cd45d0533b1965f57783f3f193168d03854/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/fc82eb6d4ebcd9e4d4769d1d51fe79d3bc7f5aa085e0b006b02d5855a6330b22/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/aaaea9275bac1a1a71471534e68cea2c0f36a0309abfcbf6350bfea88a7364a3/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bd9626af2d1f91d5d4e6765592c80fbfa0b05a4c03388b974410c26490018577/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1bb16732deaed0ae3d96ce1ee2ee3511a88be17fe24ece6cc9e5375d0a475202/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/97e36f8a83ba41579003eb2ec86623ed7af677410db4c3b7e9dde34b7d455724/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/20a4a436f9d3d46408fc0e65487a85533d0be81c149619ba237ef62fdfe9ba34/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/2229394a786406bf3b4192545e2ca4fcf98fc5c354796124b6ae275b7c36fd4a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/53261c39a3d6571f19ec4aace21aabdd4acbb1272c348c6016529e35da443393/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/77e7ae26588808b3bab31d152225ad2151c352bf1466f33de62000ee8c2abe53/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f933171ed0843735ff1e338a696a52c3b3ec8a5cd1e095688f548f45eb9ae90e/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ab79c8323539c33ff16a0089c89ef566429942efffdd415571973f859d5458c1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/988bcd68b95a71ce5109bc0100f4b4fcb440f68a3f6f60eb7da14a446e006c1a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/54f34ec3029c5c739361d8a822a49ffd45eae8fb5a82fc8c3ecda36a835c1618/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f00a74b17caf7ed19c5a2cbd1aed358eabdf6ecb4481d8528394cd7882b6775c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bbf5dfea833eb596cffb883881c1a21faf61526ebb60462225a4891998f90f01/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c8514cea86f26df124cc02de933e613fb88f3416de5bd6d6e5944eacbcfa7680/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d556a0860c50d79bbf220d94622bcab7a6f743da41a090f53b3c76ba53f8ba3c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/df232491be258ee897c0ab031c17a7f5d77b0488f57e4afdeadaa44fd4fd4576/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7beffd787d12674833d4df6e6b6e55dbbc1fffeabc08a1048eb12cb276d2d6b1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/9ae268d6544b255ad4b4405fe330f00cc52c3ef030006536c1ec7347d2dbd3f6/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/4db7e0749f956b87a4c746418b8936e237610e0cce7ab211680adaabf5796009/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/edeff9b0532b16b7b98dc221bc94400b0b8c61199c087c0471a8d91c49601228/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/188d5c12f283f6fcd45f6da6fce5b5412968f6fc36838d8672749f410ea9f552/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-63:{mountpoint:/var/lib/containers/storage/overlay/f05a182fa3701469037440af0f602059efb6ef6b56770efac8fbb1135ccfc9da/merged major:0 minor:63 fsType:overlay blockSize:0} overlay_0-67:{mountpoint:/var/lib/containers/storage/overlay/b52b8b035818d4bd9183cfbaa760d010d9369b26a123c5caca1dfd1ce2c4c176/merged major:0 minor:67 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fb0e79a51d0bc2de2b48f36242bcbb1d753bb1148cf65b7bf9d9bdcd6b388a4f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9225fe074fdb076e30c5c205a6468c4ea4d1715831a1dcce463ac2b03660e0ec/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f8ffd067a2245d64ecdd7bad2771286ff1bb13ccd71a81dc00b609db3ce195c0/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7d0603ebf7ce62fc91e9ef1c53cf460cda9a177abb7a01e9ebdc0fafba8c3a54/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/8074355e6ed07bce90e6cf32bbf67e0986d742f7c73b2bb0e2c97cc7c71edd40/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/35a05f656770da689234ef66a469c41024790a04022429b7f669369f7a041c0c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6f74f1c982325e47886e1105a7d6d1d0cfeb9d4d3fccca6454a4f0c892974785/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c3a1fdcc217754b69f4c2a168b57631fb033162e086e4194802f89a3123f3ae2/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.180662    4287 manager.go:217] Machine: {Timestamp:2026-11-02 01:24:52.180205106 +0000 UTC m=+0.053846148 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104729 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b202a86ac308c2a1f90c19d486ab5 SystemUUID:ec2b202a-86ac-308c-2a1f-90c19d486ab5 BootID:6aa6274f-7055-4b5f-ac37-69db31cb4479 Filesystems:[{Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-67 DeviceMajor:0 DeviceMinor:67 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~projected/kube-api-access-tvvr8 DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2de926-a683-49c8-9d4d-f45a09702479/volumes/kubernetes.io~projected/kube-api-access-pxns9 DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9348b2b7de5c8002e9acccaa25ccd54b4a9c0aa73e50d889c3d5e39f1726b304/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-63 DeviceMajor:0 DeviceMinor:63 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/87856a71-2d8f-47f8-8d85-4d797fb8bec4/volumes/kubernetes.io~projected/kube-api-access-txddj DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1501fcf156f5fa7938f47025ea1a96be8045f2b27bd3cb56979230a4be6f035c/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/132e89e8ef04df509abb9d5b00424e21106ab48891aa52a42b124f23d58e2a74/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volumes/kubernetes.io~projected/kube-api-access-77f27 DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0048a8958faf75002bcbc53d3b5affbd0bb3cdece1237372bc495e39b454dd8e/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/029b8e4b89efc967f55fb182cfc26eb92d8e669fa4d0685c46ccd65b850d1543/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b121f4d54e4477af976f7a35e0fbad8412c0f6208dada04f44275e5fda0c9161/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3d712a65735714262878f625e57f91c8b7a1061b2e48d852a96b2b9f974cdba/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bc310c2-330e-492f-b635-5d689f1e5203/volumes/kubernetes.io~projected/kube-api-access-vjqms DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7068729c8fb9d6c6b2e7d33c9b580b585d98a289cb68f83ed916dcc5452867c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b0586fd638a3ee62d41a31d7810c079d0a958dd8976fd2fda174951f4399d34/userdata/shm DeviceMajor:0 DeviceMinor:65 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/46964574-0965-4bf3-8ce7-1c3c5115c594/volumes/kubernetes.io~projected/kube-api-access-pwhmx DeviceMajor:0 DeviceMinor:54 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3f612b3f-b624-45ae-b05d-2b4a164fd4af/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9766361f-a2b3-4c58-9f47-899e699c2520/volumes/kubernetes.io~projected/kube-api-access-bvq2v DeviceMajor:0 DeviceMinor:53 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3464bf51-0036-4948-8bb4-43f1000d58d2/volumes/kubernetes.io~projected/kube-api-access-l7chh DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6431d4e9b3a09c4ae90e1b5e9fcad8a814c8c87103c0094892e5c955729f9b9/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0fdcd2d101d0950f75b48340425da91c9b4b6c529ab59b8f8e4f0540948f95b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1955308f-bad5-47ef-951a-95b7064ed5b2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e7237264-dbfe-4566-8e42-8ddc59360817/volumes/kubernetes.io~projected/kube-api-access-pwtcv DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0406fb90-83ff-40c9-a3aa-e834abf84e46/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03d8e0ab017b9164b69426880d82ee1f8b2e0508e63a12ed3be0bea3f06ab930/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55451071-26c3-4ebd-9377-d5cc1b967ea5/volumes/kubernetes.io~projected/kube-api-access-c2j7d DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7d20a7d0-d504-4b4d-89a6-2c48b5de6315/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03d8e0ab017b916 MacAddress:da:4d:c5:0d:37:d0 Speed:10000 Mtu:8901} {Name:132e89e8ef04df5 MacAddress:ba:89:63:9c:fa:09 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:64:ef:25:82:e9 Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:68:30:26:57:6b Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:64:ef:25:82:e9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:6d:11:7c:d0:73 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:c1:63:7c:4f:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.180886    4287 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.180926    4287 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181230    4287 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181249    4287 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-183.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181384    4287 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181392    4287 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181405    4287 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181420    4287 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181497    4287 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181681    4287 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181817    4287 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181853    4287 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 01:24:52.184752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181865    4287 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 01:24:52.185295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181875    4287 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 01:24:52.185295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181884    4287 kubelet.go:405] "Adding apiserver pod source"
Nov 02 01:24:52.185295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.181892    4287 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 01:24:52.185295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.183103    4287 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:24:52.185295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.183123    4287 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:24:52.185295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.183841    4287 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 01:24:52.186363 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186347    4287 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 01:24:52.186363 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186360    4287 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 01:24:52.186458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186447    4287 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 01:24:52.186654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186642    4287 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 01:24:52.186687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186673    4287 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 01:24:52.186766 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186758    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 01:24:52.186795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186771    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 01:24:52.186795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186778    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 01:24:52.186795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186784    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 01:24:52.186795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186789    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 01:24:52.186795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186794    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 01:24:52.186935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186799    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 01:24:52.186935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186805    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 01:24:52.186935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186812    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 01:24:52.186935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186817    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 01:24:52.186935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186826    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 01:24:52.186935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186836    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 01:24:52.186935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186858    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 01:24:52.186935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.186864    4287 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 01:24:52.190158 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.190146    4287 server.go:1263] "Started kubelet"
Nov 02 01:24:52.190255 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.190221    4287 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 01:24:52.190337 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.190243    4287 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 01:24:52.190337 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.190321    4287 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 01:24:52.190957 ip-10-0-134-183 systemd[1]: Started Kubernetes Kubelet.
Nov 02 01:24:52.192577 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.190990    4287 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 01:24:52.192577 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.191463    4287 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 01:24:52.192799 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.192781    4287 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 01:24:52.192886 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.192845    4287 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 01:24:52.192977 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.192885    4287 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-183.us-west-1.compute.internal\" not found"
Nov 02 01:24:52.192977 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.192938    4287 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 01:24:52.192977 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.192947    4287 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 01:24:52.193169 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.193052    4287 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 01:24:52.193757 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.193733    4287 factory.go:55] Registering systemd factory
Nov 02 01:24:52.193757 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.193754    4287 factory.go:223] Registration of the systemd container factory successfully
Nov 02 01:24:52.193920 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.193863    4287 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:13 +0000 UTC" deadline="2028-07-10 10:25:43.39498506 +0000 UTC"
Nov 02 01:24:52.193920 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.193888    4287 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14793h0m51.201102471s"
Nov 02 01:24:52.194104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194057    4287 factory.go:153] Registering CRI-O factory
Nov 02 01:24:52.194104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194072    4287 factory.go:223] Registration of the crio container factory successfully
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194120    4287 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194136    4287 factory.go:103] Registering Raw factory
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194149    4287 manager.go:1199] Started watching for new ooms in manager
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194189    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d20a7d0-d504-4b4d-89a6-2c48b5de6315" volumeName="kubernetes.io/projected/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-kube-api-access-tvvr8" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194218    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87856a71-2d8f-47f8-8d85-4d797fb8bec4" volumeName="kubernetes.io/projected/87856a71-2d8f-47f8-8d85-4d797fb8bec4-kube-api-access-txddj" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194242    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9766361f-a2b3-4c58-9f47-899e699c2520" volumeName="kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-multus-daemon-config" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194255    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9766361f-a2b3-4c58-9f47-899e699c2520" volumeName="kubernetes.io/projected/9766361f-a2b3-4c58-9f47-899e699c2520-kube-api-access-bvq2v" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194267    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7237264-dbfe-4566-8e42-8ddc59360817" volumeName="kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194278    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0406fb90-83ff-40c9-a3aa-e834abf84e46" volumeName="kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-config" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194289    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f612b3f-b624-45ae-b05d-2b4a164fd4af" volumeName="kubernetes.io/configmap/3f612b3f-b624-45ae-b05d-2b4a164fd4af-konnectivity-ca" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194299    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc310c2-330e-492f-b635-5d689f1e5203" volumeName="kubernetes.io/configmap/6bc310c2-330e-492f-b635-5d689f1e5203-serviceca" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194314    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bc310c2-330e-492f-b635-5d689f1e5203" volumeName="kubernetes.io/projected/6bc310c2-330e-492f-b635-5d689f1e5203-kube-api-access-vjqms" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194325    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d20a7d0-d504-4b4d-89a6-2c48b5de6315" volumeName="kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-tuned" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194336    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2de926-a683-49c8-9d4d-f45a09702479" volumeName="kubernetes.io/configmap/ba2de926-a683-49c8-9d4d-f45a09702479-iptables-alerter-script" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194347    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3464bf51-0036-4948-8bb4-43f1000d58d2" volumeName="kubernetes.io/projected/3464bf51-0036-4948-8bb4-43f1000d58d2-kube-api-access-l7chh" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194359    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f612b3f-b624-45ae-b05d-2b4a164fd4af" volumeName="kubernetes.io/secret/3f612b3f-b624-45ae-b05d-2b4a164fd4af-agent-certs" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194370    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46964574-0965-4bf3-8ce7-1c3c5115c594" volumeName="kubernetes.io/projected/46964574-0965-4bf3-8ce7-1c3c5115c594-kube-api-access-pwhmx" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194385    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87856a71-2d8f-47f8-8d85-4d797fb8bec4" volumeName="kubernetes.io/empty-dir/87856a71-2d8f-47f8-8d85-4d797fb8bec4-tmp-dir" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194396    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2de926-a683-49c8-9d4d-f45a09702479" volumeName="kubernetes.io/projected/ba2de926-a683-49c8-9d4d-f45a09702479-kube-api-access-pxns9" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194407    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0406fb90-83ff-40c9-a3aa-e834abf84e46" volumeName="kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-env-overrides" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194419    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0406fb90-83ff-40c9-a3aa-e834abf84e46" volumeName="kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 01:24:52.194420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194432    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0406fb90-83ff-40c9-a3aa-e834abf84e46" volumeName="kubernetes.io/projected/0406fb90-83ff-40c9-a3aa-e834abf84e46-kube-api-access-77f27" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194443    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46964574-0965-4bf3-8ce7-1c3c5115c594" volumeName="kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194454    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46964574-0965-4bf3-8ce7-1c3c5115c594" volumeName="kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194466    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46964574-0965-4bf3-8ce7-1c3c5115c594" volumeName="kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194481    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d20a7d0-d504-4b4d-89a6-2c48b5de6315" volumeName="kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-tmp" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194496    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9766361f-a2b3-4c58-9f47-899e699c2520" volumeName="kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-cni-binary-copy" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194511    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0406fb90-83ff-40c9-a3aa-e834abf84e46" volumeName="kubernetes.io/secret/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194550    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1955308f-bad5-47ef-951a-95b7064ed5b2" volumeName="kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194567    4287 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55451071-26c3-4ebd-9377-d5cc1b967ea5" volumeName="kubernetes.io/projected/55451071-26c3-4ebd-9377-d5cc1b967ea5-kube-api-access-c2j7d" seLinuxMountContext=""
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194581    4287 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194590    4287 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 01:24:52.196599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.194664    4287 manager.go:319] Starting recovery of all containers
Nov 02 01:24:52.212250 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.212222    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:24:52.212436 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.212411    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:24:52.228557 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.228529    4287 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 01:24:52.230064 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.230045    4287 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 01:24:52.230159 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.230073    4287 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 01:24:52.230159 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.230104    4287 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 01:24:52.230159 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.230144    4287 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 01:24:52.240281 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.240237    4287 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="9b842414db61277b1ed031fd7c12b46c218f41f5631c108fa87d08f924d42cb9" exitCode=0
Nov 02 01:24:52.240281 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.240277    4287 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="16515c20a5258e186d54f0f8d328d3075e046bb5fa1c4063b096c2fcc5376516" exitCode=0
Nov 02 01:24:52.240281 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.240284    4287 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="b3dbe5e9b4b6c5a01f2e1a1d03ad7187debaac708bf11b9d4ce55def5fee35c4" exitCode=0
Nov 02 01:24:52.240511 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.240292    4287 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="828668b816eb278f57a742165f3e58f24575b831839585842597f9a859d9f569" exitCode=0
Nov 02 01:24:52.240511 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.240297    4287 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="c965fbd6ce6d1ca806836d2d56677cb37458593575f90a607531d849ad6f9a0f" exitCode=0
Nov 02 01:24:52.240511 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.240302    4287 generic.go:356] "Generic (PLEG): container finished" podID="46964574-0965-4bf3-8ce7-1c3c5115c594" containerID="a5d53eea0d5802e8c175bcf398b77ec81f3de2c2953fd80064066594285391ee" exitCode=0
Nov 02 01:24:52.247598 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.247577    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:24:52.247928 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.247908    4287 generic.go:356] "Generic (PLEG): container finished" podID="0406fb90-83ff-40c9-a3aa-e834abf84e46" containerID="d0b3718de4cdc63735591fcbf1cbce933c314f6153c2330e42232933d82e7275" exitCode=1
Nov 02 01:24:52.252282 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.252267    4287 manager.go:324] Recovery completed
Nov 02 01:24:52.260781 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.260757    4287 generic.go:356] "Generic (PLEG): container finished" podID="0483b9f053a944bf50e838e170a69c85" containerID="a6a1f0beda2fedfaaa5a0d63439bd3328d945e89969609091b85ba408bc5b072" exitCode=0
Nov 02 01:24:52.265631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.265596    4287 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:52.266786 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.266770    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:52.266857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.266802    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:52.266857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.266815    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:52.267982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.267968    4287 cpu_manager.go:226] "Starting" policy="none"
Nov 02 01:24:52.267982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.267979    4287 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:52.268054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.267980    4287 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 01:24:52.268054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268043    4287 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 01:24:52.268198 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268185    4287 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 01:24:52.268226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268199    4287 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 01:24:52.268226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268213    4287 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 01:24:52.268226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268218    4287 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 01:24:52.268226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268224    4287 policy_none.go:50] "Start"
Nov 02 01:24:52.268332 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268233    4287 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 01:24:52.268332 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268243    4287 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 01:24:52.268393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268382    4287 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 01:24:52.268422 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268395    4287 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 01:24:52.268422 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268404    4287 policy_none.go:44] "Start"
Nov 02 01:24:52.268894 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268877    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:52.268959 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268909    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:52.268959 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.268923    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:52.272164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.272143    4287 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 01:24:52.274578 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.274565    4287 manager.go:341] "Starting Device Plugin manager"
Nov 02 01:24:52.274660 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.274588    4287 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 01:24:52.274660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.274597    4287 server.go:87] "Starting device plugin registration server"
Nov 02 01:24:52.274907 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.274893    4287 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 01:24:52.274952 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.274908    4287 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 01:24:52.275006 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.274994    4287 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 01:24:52.275111 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.275080    4287 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 01:24:52.275159 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.275115    4287 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 01:24:52.275159 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.275122    4287 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 01:24:52.275306 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.275172    4287 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:24:52.275106534Z","UUID":"15c58058-510b-4609-a370-5206fd6f7c63","Handler":null,"Name":"","Endpoint":""}
Nov 02 01:24:52.276363 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.276330    4287 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 01:24:52.276454 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.276379    4287 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-183.us-west-1.compute.internal\" not found"
Nov 02 01:24:52.276454 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.276440    4287 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 01:24:52.276454 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.276453    4287 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 01:24:52.330769 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.330278    4287 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"]
Nov 02 01:24:52.330885 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.330811    4287 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:52.331840 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.331825    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:52.331886 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.331854    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:52.331886 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.331864    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:52.332071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.332061    4287 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:52.332227 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.332211    4287 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:52.332985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.332966    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:52.333076 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333000    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:52.333076 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333012    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:52.333076 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.332970    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:52.333076 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333075    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:52.333259 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333088    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:52.333259 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.333224    4287 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-183.us-west-1.compute.internal\" not found" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.333370 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333334    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"8c8e4877a18ae50f2a48041b0698e786","Type":"ContainerStarted","Data":"d95c5a1630380ae15d91fa4caade01bc597bb2b6ac510daba69dfc143fd8ed7e"}
Nov 02 01:24:52.333427 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333389    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"8c8e4877a18ae50f2a48041b0698e786","Type":"ContainerStarted","Data":"0b0586fd638a3ee62d41a31d7810c079d0a958dd8976fd2fda174951f4399d34"}
Nov 02 01:24:52.333427 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333411    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"0483b9f053a944bf50e838e170a69c85","Type":"ContainerStarted","Data":"f3e65ce48f01aca8dde333d945503f44e05cbf73b7c980f768c7c26ccb0969c8"}
Nov 02 01:24:52.333505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333429    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"0483b9f053a944bf50e838e170a69c85","Type":"ContainerDied","Data":"a6a1f0beda2fedfaaa5a0d63439bd3328d945e89969609091b85ba408bc5b072"}
Nov 02 01:24:52.333505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333359    4287 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:52.333505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.333450    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal" event={"ID":"0483b9f053a944bf50e838e170a69c85","Type":"ContainerStarted","Data":"9348b2b7de5c8002e9acccaa25ccd54b4a9c0aa73e50d889c3d5e39f1726b304"}
Nov 02 01:24:52.334202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.334187    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:52.334256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.334218    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:52.334256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.334244    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:52.334373 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.334361    4287 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-183.us-west-1.compute.internal\" not found" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.375633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.375597    4287 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 01:24:52.376306 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.376291    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 01:24:52.376379 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.376323    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 01:24:52.376379 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.376338    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 01:24:52.376379 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.376360    4287 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.396340 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.396316    4287 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 01:24:52.443996 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.443973    4287 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:52.444085 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.444034    4287 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:52.446505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.446478    4287 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:52.493680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.493660    4287 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 01:24:52.494837 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.494823    4287 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.497015 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.496995    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.497062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.497025    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c8e4877a18ae50f2a48041b0698e786-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"8c8e4877a18ae50f2a48041b0698e786\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.497062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.497046    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.545574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.545553    4287 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.545654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.545632    4287 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.546161 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.546147    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeNotReady"
Nov 02 01:24:52.546212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.546174    4287 setters.go:546] "Node became not ready" node="ip-10-0-134-183.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T01:24:52Z","lastTransitionTime":"2026-11-02T01:24:52Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 02 01:24:52.553308 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.553291    4287 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:24:52.553389 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.553328    4287 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.553389 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.553339    4287 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.558879 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.558860    4287 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-183.us-west-1.compute.internal" is owned by different node (oldNodeID="3e93e00f-df31-4ed5-9d23-f6d43c51ce15", newNodeID=""), cleaning up...
Nov 02 01:24:52.597203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.597149    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.597203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.597176    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.597203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.597191    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c8e4877a18ae50f2a48041b0698e786-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"8c8e4877a18ae50f2a48041b0698e786\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.597337 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.597231    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c8e4877a18ae50f2a48041b0698e786-config\") pod \"kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"8c8e4877a18ae50f2a48041b0698e786\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.597337 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.597256    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.597337 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.597267    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0483b9f053a944bf50e838e170a69c85-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" (UID: \"0483b9f053a944bf50e838e170a69c85\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:52.615495 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:52.615476    4287 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:24:52.615564 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:52.615507    4287 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:53.183226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.183198    4287 apiserver.go:52] "Watching apiserver"
Nov 02 01:24:53.264772 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.264742    4287 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:53.264895 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.264861    4287 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:53.286080 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.286057    4287 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 01:24:53.286438 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.286420    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hcbh5","kube-system/konnectivity-agent-vvj9k","openshift-cluster-node-tuning-operator/tuned-7pbwr","openshift-image-registry/node-ca-9pdx5","kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal","openshift-dns/node-resolver-q77dv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-cdbt2","openshift-multus/multus-kbt65","openshift-multus/network-metrics-daemon-26vc2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj","openshift-network-diagnostics/network-check-target-wjrcp","openshift-network-operator/iptables-alerter-sf489","openshift-ovn-kubernetes/ovnkube-node-kjszn"]
Nov 02 01:24:53.288505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.288481    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:53.294356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.294338    4287 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 01:24:53.301923 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.301903    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvr8\" (UniqueName: \"kubernetes.io/projected/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-kube-api-access-tvvr8\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.301973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.301930    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-system-cni-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.301973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.301948    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-socket-dir-parent\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.301973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.301962    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxns9\" (UniqueName: \"kubernetes.io/projected/ba2de926-a683-49c8-9d4d-f45a09702479-kube-api-access-pxns9\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:53.302085 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.301976    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bc310c2-330e-492f-b635-5d689f1e5203-host\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:53.302237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302211    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqms\" (UniqueName: \"kubernetes.io/projected/6bc310c2-330e-492f-b635-5d689f1e5203-kube-api-access-vjqms\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:53.302347 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302331    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txddj\" (UniqueName: \"kubernetes.io/projected/87856a71-2d8f-47f8-8d85-4d797fb8bec4-kube-api-access-txddj\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:53.302389 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302356    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvq2v\" (UniqueName: \"kubernetes.io/projected/9766361f-a2b3-4c58-9f47-899e699c2520-kube-api-access-bvq2v\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.302389 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302373    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-dbus\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:53.302389 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302386    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-cnibin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.302519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302425    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-bin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.302519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302446    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-conf-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.302519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302464    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-etc-kubernetes\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.302519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302494    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-netns\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.302519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302516    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-bin\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.302812 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302566    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2j7d\" (UniqueName: \"kubernetes.io/projected/55451071-26c3-4ebd-9377-d5cc1b967ea5-kube-api-access-c2j7d\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:53.302812 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302653    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-kubelet\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.302812 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302701    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.302812 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302806    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-kubelet-config\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:53.303035 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302871    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f612b3f-b624-45ae-b05d-2b4a164fd4af-konnectivity-ca\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:53.303035 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302916    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-systemd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.303035 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302952    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-log-socket\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.303035 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.302990    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-system-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.303035 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303030    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-conf\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.303187 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303050    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-systemd\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.303187 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303063    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-var-lib-kubelet\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.303187 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303111    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-socket-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.303187 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303143    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-multus-daemon-config\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.303187 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303157    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2de926-a683-49c8-9d4d-f45a09702479-host-slash\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:53.303316 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303197    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysconfig\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.303316 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303241    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-host\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.303316 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303266    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.303316 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303307    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f612b3f-b624-45ae-b05d-2b4a164fd4af-agent-certs\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:53.303431 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303326    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-systemd-units\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.303431 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303351    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-var-lib-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.303431 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303412    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-etc-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.303512 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303455    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-netd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.303512 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303480    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.303512 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303505    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-device-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.303587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303544    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-multus\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.303587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303565    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-script-lib\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.303660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303623    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.303692 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303677    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7chh\" (UniqueName: \"kubernetes.io/projected/3464bf51-0036-4948-8bb4-43f1000d58d2-kube-api-access-l7chh\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.303720 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303712    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87856a71-2d8f-47f8-8d85-4d797fb8bec4-tmp-dir\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:53.303790 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.303777    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.304117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304093    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87856a71-2d8f-47f8-8d85-4d797fb8bec4-tmp-dir\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:53.304184 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304132    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-sys\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.304242 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304203    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-tuned\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.304290 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304246    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-cni-binary-copy\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.304398 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304379    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-tuned\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.304512 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304484    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-netns\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.304654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304522    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-env-overrides\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.304654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304548    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77f27\" (UniqueName: \"kubernetes.io/projected/0406fb90-83ff-40c9-a3aa-e834abf84e46-kube-api-access-77f27\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.304654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304576    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-run\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.304654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304625    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-registration-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.304861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304653    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.304861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304686    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-k8s-cni-cncf-io\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.304861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304718    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovn-node-metrics-cert\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.304861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304759    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-tmp\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.304861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304792    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:53.304861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304826    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-kubelet\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.304861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304852    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-slash\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304883    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-node-log\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304918    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304953    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-hostroot\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.304986    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-cnibin\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305016    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-binary-copy\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305052    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305087    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305120    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:53.305174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305154    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba2de926-a683-49c8-9d4d-f45a09702479-iptables-alerter-script\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305179    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87856a71-2d8f-47f8-8d85-4d797fb8bec4-hosts-file\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305223    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhmx\" (UniqueName: \"kubernetes.io/projected/46964574-0965-4bf3-8ce7-1c3c5115c594-kube-api-access-pwhmx\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305250    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305282    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-sys-fs\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305316    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-ovn\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305348    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-config\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305374    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-kubernetes\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305406    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-lib-modules\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305438    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-multus-certs\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305476    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-modprobe-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305509    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bc310c2-330e-492f-b635-5d689f1e5203-serviceca\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:53.305583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.305537    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.306124 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.306080    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-tmp\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.325777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.325756    4287 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:24:53.325860 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:53.325792    4287 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:53.326597 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.326584    4287 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 01:24:53.326659 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:24:53.326630    4287 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-183.us-west-1.compute.internal"
Nov 02 01:24:53.344239 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344212    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 01:24:53.344497 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344483    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 01:24:53.344497 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344481    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 01:24:53.344644 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344481    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 01:24:53.344644 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344548    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 01:24:53.344808 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344790    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 01:24:53.344862 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344805    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 01:24:53.344862 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344844    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 01:24:53.344931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.344908    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 01:24:53.345036 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345021    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:24:53.345267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345252    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:24:53.345310 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345271    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 01:24:53.345534 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345520    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 01:24:53.345594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345572    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 01:24:53.345670 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345635    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwsn\""
Nov 02 01:24:53.345809 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345795    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 01:24:53.345869 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345802    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 01:24:53.345869 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345815    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 01:24:53.345869 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345846    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfbzc\""
Nov 02 01:24:53.345869 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345867    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345869    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345886    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kv2jl\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345892    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345886    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gh69v\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345903    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345925    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5zpq\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345931    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6qqtd\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345942    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345954    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345978    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wwr4r\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345982    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pw4q8\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.345995    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gsg88\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346007    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346000    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346023    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346037    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82k56\""
Nov 02 01:24:53.346041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346045    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2dvkb\""
Nov 02 01:24:53.346480 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346038    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qtccq\""
Nov 02 01:24:53.346480 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346423    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1955308f-bad5-47ef-951a-95b7064ed5b2-original-pull-secret\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:53.346673 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346659    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.346712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346681    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-binary-copy\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.346840 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346826    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba2de926-a683-49c8-9d4d-f45a09702479-iptables-alerter-script\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:53.346978 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.346966    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-config\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.347379 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.347362    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6bc310c2-330e-492f-b635-5d689f1e5203-serviceca\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:53.353415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.353400    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f612b3f-b624-45ae-b05d-2b4a164fd4af-konnectivity-ca\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:53.353530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.353517    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-multus-daemon-config\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.353560 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.353549    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f612b3f-b624-45ae-b05d-2b4a164fd4af-agent-certs\") pod \"konnectivity-agent-vvj9k\" (UID: \"3f612b3f-b624-45ae-b05d-2b4a164fd4af\") " pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:53.353671 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.353657    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/46964574-0965-4bf3-8ce7-1c3c5115c594-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.354706 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.354692    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9766361f-a2b3-4c58-9f47-899e699c2520-cni-binary-copy\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.373938 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.373912    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqms\" (UniqueName: \"kubernetes.io/projected/6bc310c2-330e-492f-b635-5d689f1e5203-kube-api-access-vjqms\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:53.374215 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.374188    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxns9\" (UniqueName: \"kubernetes.io/projected/ba2de926-a683-49c8-9d4d-f45a09702479-kube-api-access-pxns9\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:53.374323 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.374301    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhmx\" (UniqueName: \"kubernetes.io/projected/46964574-0965-4bf3-8ce7-1c3c5115c594-kube-api-access-pwhmx\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.374416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.374329    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7chh\" (UniqueName: \"kubernetes.io/projected/3464bf51-0036-4948-8bb4-43f1000d58d2-kube-api-access-l7chh\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.374416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.374341    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txddj\" (UniqueName: \"kubernetes.io/projected/87856a71-2d8f-47f8-8d85-4d797fb8bec4-kube-api-access-txddj\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:53.374416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.374345    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvr8\" (UniqueName: \"kubernetes.io/projected/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-kube-api-access-tvvr8\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.374537 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.374463    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2j7d\" (UniqueName: \"kubernetes.io/projected/55451071-26c3-4ebd-9377-d5cc1b967ea5-kube-api-access-c2j7d\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:53.374575 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.374553    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvq2v\" (UniqueName: \"kubernetes.io/projected/9766361f-a2b3-4c58-9f47-899e699c2520-kube-api-access-bvq2v\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.374665 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.374650    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtcv\" (UniqueName: \"kubernetes.io/projected/e7237264-dbfe-4566-8e42-8ddc59360817-kube-api-access-pwtcv\") pod \"network-check-target-wjrcp\" (UID: \"e7237264-dbfe-4566-8e42-8ddc59360817\") " pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:53.387364 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.387346    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 01:24:53.396790 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.396774    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovn-node-metrics-cert\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.405837 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405805    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-kubelet-config\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:53.405945 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405841    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-systemd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.405945 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405865    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-log-socket\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.405945 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405888    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-system-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.405945 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405912    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-conf\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.405945 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405932    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-systemd\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.405945 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405930    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-log-socket\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405955    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-var-lib-kubelet\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405937    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-systemd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405983    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-socket-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405998    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-system-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406016    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-var-lib-kubelet\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406008    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2de926-a683-49c8-9d4d-f45a09702479-host-slash\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406055    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysconfig\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.405942    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-kubelet-config\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406084    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-socket-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406092    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba2de926-a683-49c8-9d4d-f45a09702479-host-slash\") pod \"iptables-alerter-sf489\" (UID: \"ba2de926-a683-49c8-9d4d-f45a09702479\") " pod="openshift-network-operator/iptables-alerter-sf489"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406097    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-conf\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406081    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-host\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406138    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysconfig\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406054    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-systemd\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406154    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-host\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406159    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406193    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-systemd-units\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.406216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406216    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-var-lib-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406246    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-etc-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406195    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406270    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-netd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406274    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-systemd-units\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406291    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-device-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406301    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-var-lib-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406314    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-multus\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406336    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406322    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-etc-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406362    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406381    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-sys\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406391    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-multus\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406402    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-netns\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406423    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-run\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406446    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-registration-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406455    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-device-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406466    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406487    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-k8s-cni-cncf-io\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406505    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-sys\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406512    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-kubelet\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406535    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-slash\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406557    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-node-log\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406580    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406633    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-hostroot\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406657    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-cnibin\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406661    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-sysctl-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406682    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-k8s-cni-cncf-io\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406684    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406707    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-node-log\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406728    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87856a71-2d8f-47f8-8d85-4d797fb8bec4-hosts-file\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406752    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-kubelet\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406756    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406782    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406789    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-sys-fs\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406803    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406832    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-ovn\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406840    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-sys-fs\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406832    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-cnibin\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406731    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406870    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-hostroot\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406880    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-ovn\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406890    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-cni-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406891    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-kubernetes\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406904    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-netns\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406919    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-lib-modules\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406926    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/87856a71-2d8f-47f8-8d85-4d797fb8bec4-hosts-file\") pod \"node-resolver-q77dv\" (UID: \"87856a71-2d8f-47f8-8d85-4d797fb8bec4\") " pod="openshift-dns/node-resolver-q77dv"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406929    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-kubernetes\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406930    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-run\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406867    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-slash\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406942    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-multus-certs\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406980    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-modprobe-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406993    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3464bf51-0036-4948-8bb4-43f1000d58d2-registration-dir\") pod \"aws-ebs-csi-driver-node-fshnj\" (UID: \"3464bf51-0036-4948-8bb4-43f1000d58d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fshnj"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406996    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407018    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-run-openvswitch\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407041    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-lib-modules\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407031    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-system-cni-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407059    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-run-multus-certs\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407079    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-socket-dir-parent\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407083    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46964574-0965-4bf3-8ce7-1c3c5115c594-system-cni-dir\") pod \"multus-additional-cni-plugins-cdbt2\" (UID: \"46964574-0965-4bf3-8ce7-1c3c5115c594\") " pod="openshift-multus/multus-additional-cni-plugins-cdbt2"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407092    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7d20a7d0-d504-4b4d-89a6-2c48b5de6315-etc-modprobe-d\") pod \"tuned-7pbwr\" (UID: \"7d20a7d0-d504-4b4d-89a6-2c48b5de6315\") " pod="openshift-cluster-node-tuning-operator/tuned-7pbwr"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407097    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bc310c2-330e-492f-b635-5d689f1e5203-host\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407117    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6bc310c2-330e-492f-b635-5d689f1e5203-host\") pod \"node-ca-9pdx5\" (UID: \"6bc310c2-330e-492f-b635-5d689f1e5203\") " pod="openshift-image-registry/node-ca-9pdx5"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407119    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-dbus\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407134    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-cnibin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.406339    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-netd\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407130    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-socket-dir-parent\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407150    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-bin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407165    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-conf-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407170    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-cni-bin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407172    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1955308f-bad5-47ef-951a-95b7064ed5b2-dbus\") pod \"global-pull-secret-syncer-hcbh5\" (UID: \"1955308f-bad5-47ef-951a-95b7064ed5b2\") " pod="kube-system/global-pull-secret-syncer-hcbh5"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407178    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-etc-kubernetes\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407188    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-cnibin\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407209    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-etc-kubernetes\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.407164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407209    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-multus-conf-dir\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407245    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-netns\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407287    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-bin\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407294    4287 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407315    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-kubelet\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407342    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407351    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-netns\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407350    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-cni-bin\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407364    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9766361f-a2b3-4c58-9f47-899e699c2520-host-var-lib-kubelet\") pod \"multus-kbt65\" (UID: \"9766361f-a2b3-4c58-9f47-899e699c2520\") " pod="openshift-multus/multus-kbt65"
Nov 02 01:24:53.409342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.407394    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0406fb90-83ff-40c9-a3aa-e834abf84e46-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.409572 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.409461    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55451071-26c3-4ebd-9377-d5cc1b967ea5-metrics-certs\") pod \"network-metrics-daemon-26vc2\" (UID: \"55451071-26c3-4ebd-9377-d5cc1b967ea5\") " pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:53.426741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.426718    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 01:24:53.427025 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.427010    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 01:24:53.427198 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.427176    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 01:24:53.427416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.427400    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 01:24:53.433921 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.433871    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77f27\" (UniqueName: \"kubernetes.io/projected/0406fb90-83ff-40c9-a3aa-e834abf84e46-kube-api-access-77f27\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.434109 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.434091    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-ovnkube-script-lib\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.436776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.436749    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0406fb90-83ff-40c9-a3aa-e834abf84e46-env-overrides\") pod \"ovnkube-node-kjszn\" (UID: \"0406fb90-83ff-40c9-a3aa-e834abf84e46\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.589590 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.589559    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-26vc2"
Nov 02 01:24:53.778868 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.778844    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:53.994359 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:53.994317    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-26vc2"]
Nov 02 01:24:53.997958 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:24:53.997923    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55451071_26c3_4ebd_9377_d5cc1b967ea5.slice/crio-d54c3f0ae8d842d0abdbaee0dd545a65e2c9dab7116e29c4621a020a1d2a7683 WatchSource:0}: Error finding container d54c3f0ae8d842d0abdbaee0dd545a65e2c9dab7116e29c4621a020a1d2a7683: Status 404 returned error can't find the container with id d54c3f0ae8d842d0abdbaee0dd545a65e2c9dab7116e29c4621a020a1d2a7683
Nov 02 01:24:54.000340 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:54.000321    4287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:24:54.269407 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:54.269368    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-26vc2" event={"ID":"55451071-26c3-4ebd-9377-d5cc1b967ea5","Type":"ContainerStarted","Data":"d54c3f0ae8d842d0abdbaee0dd545a65e2c9dab7116e29c4621a020a1d2a7683"}
Nov 02 01:24:56.275145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:56.275101    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-26vc2" event={"ID":"55451071-26c3-4ebd-9377-d5cc1b967ea5","Type":"ContainerStarted","Data":"0fc71c8a976b931446a45b3bdc7a1682b930a19eeb05ce8c0a4ba3d50ca9d76f"}
Nov 02 01:24:56.275562 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:56.275154    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-26vc2" event={"ID":"55451071-26c3-4ebd-9377-d5cc1b967ea5","Type":"ContainerStarted","Data":"65985301d7c810ddfb55dda668bc35795afeca72f5b11b4982e2611402fcc684"}
Nov 02 01:24:57.759089 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:57.759058    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vvj9k"
Nov 02 01:24:58.526360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:58.526325    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:24:59.004413 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:59.004329    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:24:59.004762 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:24:59.004541    4287 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 01:25:02.891206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:02.891169    4287 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-183.us-west-1.compute.internal" event="NodeReady"
Nov 02 01:25:03.028392 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.028353    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c5g5c"]
Nov 02 01:25:03.030046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.030030    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.050415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.050383    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8zv9z"]
Nov 02 01:25:03.051957 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.051940    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zv9z"
Nov 02 01:25:03.062820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.062796    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbb3db6-489e-4639-a9f2-91c9516472af-metrics-tls\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.062942 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.062833    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8q9\" (UniqueName: \"kubernetes.io/projected/ecbb3db6-489e-4639-a9f2-91c9516472af-kube-api-access-dv8q9\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.062942 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.062854    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbb3db6-489e-4639-a9f2-91c9516472af-tmp-dir\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.063027 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.062961    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbb3db6-489e-4639-a9f2-91c9516472af-config-volume\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.115078 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.115014    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 01:25:03.115078 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.115021    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tl75j\""
Nov 02 01:25:03.115078 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.115083    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q5j6l\""
Nov 02 01:25:03.115633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.115547    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c5g5c"]
Nov 02 01:25:03.117039 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.117014    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 01:25:03.120665 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.120645    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 01:25:03.121388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.121368    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zv9z"]
Nov 02 01:25:03.136691 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.136658    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-29gbz"]
Nov 02 01:25:03.138464 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.138449    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.163520 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163484    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbb3db6-489e-4639-a9f2-91c9516472af-metrics-tls\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.163733 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163534    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46bc54e2-ab02-440f-a4f8-58054df61524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.163733 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163564    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44945\" (UniqueName: \"kubernetes.io/projected/46bc54e2-ab02-440f-a4f8-58054df61524-kube-api-access-44945\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.163733 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163589    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46bc54e2-ab02-440f-a4f8-58054df61524-data-volume\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.163733 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163660    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46bc54e2-ab02-440f-a4f8-58054df61524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.163878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163740    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8q9\" (UniqueName: \"kubernetes.io/projected/ecbb3db6-489e-4639-a9f2-91c9516472af-kube-api-access-dv8q9\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.163878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163761    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4145c15-e701-4783-bffb-10e26a0fc796-cert\") pod \"ingress-canary-8zv9z\" (UID: \"c4145c15-e701-4783-bffb-10e26a0fc796\") " pod="openshift-ingress-canary/ingress-canary-8zv9z"
Nov 02 01:25:03.163878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163782    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbb3db6-489e-4639-a9f2-91c9516472af-tmp-dir\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.163878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163799    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lld\" (UniqueName: \"kubernetes.io/projected/c4145c15-e701-4783-bffb-10e26a0fc796-kube-api-access-f8lld\") pod \"ingress-canary-8zv9z\" (UID: \"c4145c15-e701-4783-bffb-10e26a0fc796\") " pod="openshift-ingress-canary/ingress-canary-8zv9z"
Nov 02 01:25:03.163878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163855    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46bc54e2-ab02-440f-a4f8-58054df61524-crio-socket\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.164082 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.163900    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbb3db6-489e-4639-a9f2-91c9516472af-config-volume\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.164175 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.164152    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecbb3db6-489e-4639-a9f2-91c9516472af-tmp-dir\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.164392 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.164371    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbb3db6-489e-4639-a9f2-91c9516472af-config-volume\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.166019 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.165998    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ecbb3db6-489e-4639-a9f2-91c9516472af-metrics-tls\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.207950 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.207913    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l67lg\""
Nov 02 01:25:03.208115 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.207972    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 01:25:03.216198 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.216168    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 01:25:03.216310 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.216260    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 01:25:03.216356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.216331    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 01:25:03.216388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.216350    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 01:25:03.216388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.216374    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 01:25:03.223754 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.223724    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-29gbz"]
Nov 02 01:25:03.224859 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.224837    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8q9\" (UniqueName: \"kubernetes.io/projected/ecbb3db6-489e-4639-a9f2-91c9516472af-kube-api-access-dv8q9\") pod \"dns-default-c5g5c\" (UID: \"ecbb3db6-489e-4639-a9f2-91c9516472af\") " pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.265078 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265038    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46bc54e2-ab02-440f-a4f8-58054df61524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.265078 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265075    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44945\" (UniqueName: \"kubernetes.io/projected/46bc54e2-ab02-440f-a4f8-58054df61524-kube-api-access-44945\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.265329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265104    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46bc54e2-ab02-440f-a4f8-58054df61524-data-volume\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.265329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265121    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46bc54e2-ab02-440f-a4f8-58054df61524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.265329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265159    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4145c15-e701-4783-bffb-10e26a0fc796-cert\") pod \"ingress-canary-8zv9z\" (UID: \"c4145c15-e701-4783-bffb-10e26a0fc796\") " pod="openshift-ingress-canary/ingress-canary-8zv9z"
Nov 02 01:25:03.265329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265187    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lld\" (UniqueName: \"kubernetes.io/projected/c4145c15-e701-4783-bffb-10e26a0fc796-kube-api-access-f8lld\") pod \"ingress-canary-8zv9z\" (UID: \"c4145c15-e701-4783-bffb-10e26a0fc796\") " pod="openshift-ingress-canary/ingress-canary-8zv9z"
Nov 02 01:25:03.265329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265211    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46bc54e2-ab02-440f-a4f8-58054df61524-crio-socket\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.265329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265283    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46bc54e2-ab02-440f-a4f8-58054df61524-crio-socket\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.265602 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265496    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46bc54e2-ab02-440f-a4f8-58054df61524-data-volume\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.265793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.265768    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46bc54e2-ab02-440f-a4f8-58054df61524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.267696 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.267674    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46bc54e2-ab02-440f-a4f8-58054df61524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.267904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.267882    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4145c15-e701-4783-bffb-10e26a0fc796-cert\") pod \"ingress-canary-8zv9z\" (UID: \"c4145c15-e701-4783-bffb-10e26a0fc796\") " pod="openshift-ingress-canary/ingress-canary-8zv9z"
Nov 02 01:25:03.326105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.326071    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44945\" (UniqueName: \"kubernetes.io/projected/46bc54e2-ab02-440f-a4f8-58054df61524-kube-api-access-44945\") pod \"insights-runtime-extractor-29gbz\" (UID: \"46bc54e2-ab02-440f-a4f8-58054df61524\") " pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.326256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.326236    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lld\" (UniqueName: \"kubernetes.io/projected/c4145c15-e701-4783-bffb-10e26a0fc796-kube-api-access-f8lld\") pod \"ingress-canary-8zv9z\" (UID: \"c4145c15-e701-4783-bffb-10e26a0fc796\") " pod="openshift-ingress-canary/ingress-canary-8zv9z"
Nov 02 01:25:03.341115 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.341082    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:03.361485 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.361443    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zv9z"
Nov 02 01:25:03.449348 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.449316    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-29gbz"
Nov 02 01:25:03.711063 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.710978    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c5g5c"]
Nov 02 01:25:03.714449 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:25:03.714408    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbb3db6_489e_4639_a9f2_91c9516472af.slice/crio-d6c5c527a6eb7675a82e47ba8dbd2294a327e29776cea34a84e22198d9e0d5cb WatchSource:0}: Error finding container d6c5c527a6eb7675a82e47ba8dbd2294a327e29776cea34a84e22198d9e0d5cb: Status 404 returned error can't find the container with id d6c5c527a6eb7675a82e47ba8dbd2294a327e29776cea34a84e22198d9e0d5cb
Nov 02 01:25:03.758942 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.758910    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zv9z"]
Nov 02 01:25:03.762124 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:25:03.762100    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4145c15_e701_4783_bffb_10e26a0fc796.slice/crio-8d24a624d70ce047837f11b61108ae42b091a94d1ecb459fc3e6094d4dc1ca6e WatchSource:0}: Error finding container 8d24a624d70ce047837f11b61108ae42b091a94d1ecb459fc3e6094d4dc1ca6e: Status 404 returned error can't find the container with id 8d24a624d70ce047837f11b61108ae42b091a94d1ecb459fc3e6094d4dc1ca6e
Nov 02 01:25:03.818387 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:03.818353    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-29gbz"]
Nov 02 01:25:03.821968 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:25:03.821940    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bc54e2_ab02_440f_a4f8_58054df61524.slice/crio-339b616c587f6f1c8c4c0288c8a2d9d50ea6497cc343c915b15f10be7a202689 WatchSource:0}: Error finding container 339b616c587f6f1c8c4c0288c8a2d9d50ea6497cc343c915b15f10be7a202689: Status 404 returned error can't find the container with id 339b616c587f6f1c8c4c0288c8a2d9d50ea6497cc343c915b15f10be7a202689
Nov 02 01:25:04.292865 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:04.292826    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zv9z" event={"ID":"c4145c15-e701-4783-bffb-10e26a0fc796","Type":"ContainerStarted","Data":"8d24a624d70ce047837f11b61108ae42b091a94d1ecb459fc3e6094d4dc1ca6e"}
Nov 02 01:25:04.293940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:04.293916    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c5g5c" event={"ID":"ecbb3db6-489e-4639-a9f2-91c9516472af","Type":"ContainerStarted","Data":"d6c5c527a6eb7675a82e47ba8dbd2294a327e29776cea34a84e22198d9e0d5cb"}
Nov 02 01:25:04.295271 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:04.295248    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-29gbz" event={"ID":"46bc54e2-ab02-440f-a4f8-58054df61524","Type":"ContainerStarted","Data":"c194ed7b9f7eb791f94875229a900b708f7be907f50ae97e86f281a249512a09"}
Nov 02 01:25:04.295357 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:04.295291    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-29gbz" event={"ID":"46bc54e2-ab02-440f-a4f8-58054df61524","Type":"ContainerStarted","Data":"339b616c587f6f1c8c4c0288c8a2d9d50ea6497cc343c915b15f10be7a202689"}
Nov 02 01:25:05.300255 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:05.300209    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-29gbz" event={"ID":"46bc54e2-ab02-440f-a4f8-58054df61524","Type":"ContainerStarted","Data":"9082897dd15e2e5e135c0083f6f5d2240f9157295db2db006298faf2c38fdddc"}
Nov 02 01:25:06.306539 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.306500    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c5g5c" event={"ID":"ecbb3db6-489e-4639-a9f2-91c9516472af","Type":"ContainerStarted","Data":"302554b1fb9f9a0c5c00891380cd60d941d438ee6c18a80a360f78c71104e58f"}
Nov 02 01:25:06.734340 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.733734    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"]
Nov 02 01:25:06.739672 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.739222    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.792209 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.792177    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.792391 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.792237    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.792391 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.792365    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.792525 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.792400    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgszd\" (UniqueName: \"kubernetes.io/projected/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-kube-api-access-rgszd\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.795626 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.795529    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 01:25:06.796007 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.795985    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 01:25:06.797097 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.797042    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sxcmf\""
Nov 02 01:25:06.841232 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.841197    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 01:25:06.841232 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.841208    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 01:25:06.841470 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.841316    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 01:25:06.842172 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.842148    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"]
Nov 02 01:25:06.893380 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.893338    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.893560 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.893393    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.893560 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.893420    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.893560 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.893437    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgszd\" (UniqueName: \"kubernetes.io/projected/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-kube-api-access-rgszd\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.894273 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.894181    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.896402 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.896375    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.896649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.896600    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:06.958084 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:06.958048    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgszd\" (UniqueName: \"kubernetes.io/projected/f92ba5d0-6d38-44d5-a41f-8d14274ca4f2-kube-api-access-rgszd\") pod \"prometheus-operator-9cd89b74c-vmxbx\" (UID: \"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:07.053833 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.053801    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"
Nov 02 01:25:07.311864 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.311766    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zv9z" event={"ID":"c4145c15-e701-4783-bffb-10e26a0fc796","Type":"ContainerStarted","Data":"240d08e88a55b5793143a9a3246ddb51bea805967e23b4777db23616bc1c6c73"}
Nov 02 01:25:07.313747 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.313717    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c5g5c" event={"ID":"ecbb3db6-489e-4639-a9f2-91c9516472af","Type":"ContainerStarted","Data":"fb116878dde6647309e2ae1dd0c7afc52dcea7ad37b2fa82b19f68a1dc7f89c6"}
Nov 02 01:25:07.313902 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.313791    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:07.316006 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.315985    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-29gbz" event={"ID":"46bc54e2-ab02-440f-a4f8-58054df61524","Type":"ContainerStarted","Data":"260f689c22dbd8ab5272f225c2e4ce984985dc129b55eb00d29fdb8daf5303b2"}
Nov 02 01:25:07.433294 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.433233    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8zv9z" podStartSLOduration=2.077731064 podStartE2EDuration="5.433218387s" totalImagesPullingTime="3.355487323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:07.432767478 +0000 UTC m=+15.306408523" watchObservedRunningTime="2026-11-02 01:25:07.433218387 +0000 UTC m=+15.306859459"
Nov 02 01:25:07.436307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.436280    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx"]
Nov 02 01:25:07.439559 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:25:07.439530    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92ba5d0_6d38_44d5_a41f_8d14274ca4f2.slice/crio-8c988e922bf11691dbf53f059706dc6d0885fdf4f9db2096b3bbff255e9fb154 WatchSource:0}: Error finding container 8c988e922bf11691dbf53f059706dc6d0885fdf4f9db2096b3bbff255e9fb154: Status 404 returned error can't find the container with id 8c988e922bf11691dbf53f059706dc6d0885fdf4f9db2096b3bbff255e9fb154
Nov 02 01:25:07.565233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.565130    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-c5g5c" podStartSLOduration=3.122791914 podStartE2EDuration="5.56511533s" totalImagesPullingTime="2.442323416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:07.564236314 +0000 UTC m=+15.437877368" watchObservedRunningTime="2026-11-02 01:25:07.56511533 +0000 UTC m=+15.438756379"
Nov 02 01:25:07.683408 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:07.683346    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-29gbz" podStartSLOduration=1.560674418 podStartE2EDuration="4.683328785s" totalImagesPullingTime="3.122654367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:25:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:07.682481045 +0000 UTC m=+15.556122092" watchObservedRunningTime="2026-11-02 01:25:07.683328785 +0000 UTC m=+15.556969838"
Nov 02 01:25:08.319757 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:08.319717    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx" event={"ID":"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2","Type":"ContainerStarted","Data":"8c988e922bf11691dbf53f059706dc6d0885fdf4f9db2096b3bbff255e9fb154"}
Nov 02 01:25:09.347982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:09.347947    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjszn"
Nov 02 01:25:10.329117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:10.329082    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx" event={"ID":"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2","Type":"ContainerStarted","Data":"423eafb987297e279bc7f2e7b27c01b1d6595f130225be0b2543d563043650f5"}
Nov 02 01:25:10.329267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:10.329133    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx" event={"ID":"f92ba5d0-6d38-44d5-a41f-8d14274ca4f2","Type":"ContainerStarted","Data":"b6fc8b2667c9d889ab99ad7398e8539bb3c6a6f69f9e05954cf0c6fbe1fe7f0e"}
Nov 02 01:25:10.447314 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:10.447265    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-vmxbx" podStartSLOduration=2.42329997 podStartE2EDuration="4.447251394s" totalImagesPullingTime="2.023951424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:10.446738153 +0000 UTC m=+18.320379230" watchObservedRunningTime="2026-11-02 01:25:10.447251394 +0000 UTC m=+18.320892496"
Nov 02 01:25:13.517779 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.517749    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xldxx"]
Nov 02 01:25:13.519751 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.519733    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.583799 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.583771    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 01:25:13.639955 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.639916    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-textfile\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.639955 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.639954    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-sys\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.640157 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.639972    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f93bc86-436b-4eec-a84c-b391416a9844-metrics-client-ca\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.640157 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.639990    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.640157 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.640095    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-root\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.640157 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.640145    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwwt4\" (UniqueName: \"kubernetes.io/projected/2f93bc86-436b-4eec-a84c-b391416a9844-kube-api-access-rwwt4\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.640273 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.640176    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-tls\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.640273 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.640198    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-wtmp\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.640273 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.640245    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-accelerators-collector-config\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.683399 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.683364    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 01:25:13.683547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.683461    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4dgbb\""
Nov 02 01:25:13.704295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.704269    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 01:25:13.740618 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740587    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-tls\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740642    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-wtmp\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740665    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-accelerators-collector-config\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740709    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-textfile\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740729    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-sys\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740751    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f93bc86-436b-4eec-a84c-b391416a9844-metrics-client-ca\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740774    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740805    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-root\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740845    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-wtmp\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740845    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-sys\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740855    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f93bc86-436b-4eec-a84c-b391416a9844-root\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.740930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.740905    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwwt4\" (UniqueName: \"kubernetes.io/projected/2f93bc86-436b-4eec-a84c-b391416a9844-kube-api-access-rwwt4\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.741198 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.741136    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-textfile\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.741476 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.741451    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-accelerators-collector-config\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.741535 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.741465    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f93bc86-436b-4eec-a84c-b391416a9844-metrics-client-ca\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.743187 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.743168    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-tls\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.743264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.743214    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f93bc86-436b-4eec-a84c-b391416a9844-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.803861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.803784    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwwt4\" (UniqueName: \"kubernetes.io/projected/2f93bc86-436b-4eec-a84c-b391416a9844-kube-api-access-rwwt4\") pod \"node-exporter-xldxx\" (UID: \"2f93bc86-436b-4eec-a84c-b391416a9844\") " pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.830882 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:13.830844    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xldxx"
Nov 02 01:25:13.840419 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:25:13.840389    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f93bc86_436b_4eec_a84c_b391416a9844.slice/crio-4c69d2bfa15f091cd2c2b7333d0fd9d898ee8452bed4ff696134400f2fc68bda WatchSource:0}: Error finding container 4c69d2bfa15f091cd2c2b7333d0fd9d898ee8452bed4ff696134400f2fc68bda: Status 404 returned error can't find the container with id 4c69d2bfa15f091cd2c2b7333d0fd9d898ee8452bed4ff696134400f2fc68bda
Nov 02 01:25:14.342772 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:14.342736    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xldxx" event={"ID":"2f93bc86-436b-4eec-a84c-b391416a9844","Type":"ContainerStarted","Data":"4c69d2bfa15f091cd2c2b7333d0fd9d898ee8452bed4ff696134400f2fc68bda"}
Nov 02 01:25:16.038164 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:25:16.038129    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:25:16.351138 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:16.351058    4287 generic.go:356] "Generic (PLEG): container finished" podID="2f93bc86-436b-4eec-a84c-b391416a9844" containerID="df7eeccb5cef7e29ccd0902a634e9e283c5bcdb3a53876dd403d83fda7421214" exitCode=0
Nov 02 01:25:16.351138 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:16.351126    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xldxx" event={"ID":"2f93bc86-436b-4eec-a84c-b391416a9844","Type":"ContainerDied","Data":"df7eeccb5cef7e29ccd0902a634e9e283c5bcdb3a53876dd403d83fda7421214"}
Nov 02 01:25:17.321840 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:17.321808    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c5g5c"
Nov 02 01:25:17.356165 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:17.356133    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xldxx" event={"ID":"2f93bc86-436b-4eec-a84c-b391416a9844","Type":"ContainerStarted","Data":"931eb115f4d017432f4d55b94b95cb85ae2d57c59026bf437e704f536f249a73"}
Nov 02 01:25:17.356315 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:17.356188    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xldxx" event={"ID":"2f93bc86-436b-4eec-a84c-b391416a9844","Type":"ContainerStarted","Data":"2bfd068eac50c3aff5cd634a05307a233b365651ce1e9a949f69be262f78c238"}
Nov 02 01:25:17.556335 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:17.556282    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xldxx" podStartSLOduration=2.931084855 podStartE2EDuration="4.55626887s" totalImagesPullingTime="1.55243754s" totalInitContainerRuntime="72.746475ms" isStatefulPod=true podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:17.555738532 +0000 UTC m=+25.429379587" watchObservedRunningTime="2026-11-02 01:25:17.55626887 +0000 UTC m=+25.429909923"
Nov 02 01:25:18.115838 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.115805    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"]
Nov 02 01:25:18.118984 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.118968    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:18.171941 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.171917    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgnsk\" (UniqueName: \"kubernetes.io/projected/8e58fc7d-197a-4856-8f1c-e0787c11ab2c-kube-api-access-zgnsk\") pod \"monitoring-plugin-77f4f76594-p9wf7\" (UID: \"8e58fc7d-197a-4856-8f1c-e0787c11ab2c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:18.172064 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.171960    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e58fc7d-197a-4856-8f1c-e0787c11ab2c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p9wf7\" (UID: \"8e58fc7d-197a-4856-8f1c-e0787c11ab2c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:18.172427 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.172407    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 01:25:18.172519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.172493    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ztrsg\""
Nov 02 01:25:18.178044 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.178025    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"]
Nov 02 01:25:18.273209 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.273175    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgnsk\" (UniqueName: \"kubernetes.io/projected/8e58fc7d-197a-4856-8f1c-e0787c11ab2c-kube-api-access-zgnsk\") pod \"monitoring-plugin-77f4f76594-p9wf7\" (UID: \"8e58fc7d-197a-4856-8f1c-e0787c11ab2c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:18.273373 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.273216    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e58fc7d-197a-4856-8f1c-e0787c11ab2c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p9wf7\" (UID: \"8e58fc7d-197a-4856-8f1c-e0787c11ab2c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:18.275734 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.275713    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e58fc7d-197a-4856-8f1c-e0787c11ab2c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p9wf7\" (UID: \"8e58fc7d-197a-4856-8f1c-e0787c11ab2c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:18.334148 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.334113    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgnsk\" (UniqueName: \"kubernetes.io/projected/8e58fc7d-197a-4856-8f1c-e0787c11ab2c-kube-api-access-zgnsk\") pod \"monitoring-plugin-77f4f76594-p9wf7\" (UID: \"8e58fc7d-197a-4856-8f1c-e0787c11ab2c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:18.429931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.429859    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:18.798185 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:18.798156    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"]
Nov 02 01:25:18.801182 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:25:18.801154    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e58fc7d_197a_4856_8f1c_e0787c11ab2c.slice/crio-59b59aebef859f9827e8553b3d5a9daba28cd3e32db942a67b17c6c247fe062d WatchSource:0}: Error finding container 59b59aebef859f9827e8553b3d5a9daba28cd3e32db942a67b17c6c247fe062d: Status 404 returned error can't find the container with id 59b59aebef859f9827e8553b3d5a9daba28cd3e32db942a67b17c6c247fe062d
Nov 02 01:25:19.363625 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:19.363579    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7" event={"ID":"8e58fc7d-197a-4856-8f1c-e0787c11ab2c","Type":"ContainerStarted","Data":"59b59aebef859f9827e8553b3d5a9daba28cd3e32db942a67b17c6c247fe062d"}
Nov 02 01:25:20.141620 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.141572    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d9b68d5bf-crfsr"]
Nov 02 01:25:20.144763 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.144736    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.186304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.186270    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-serving-cert\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.186458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.186316    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-service-ca\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.186536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.186451    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-oauth-config\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.186536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.186481    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-console-config\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.186536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.186525    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3316930a-c6d8-48fa-8768-4638b2291be5-tmp\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.186703 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.186581    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-trusted-ca-bundle\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.186749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.186718    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjsh\" (UniqueName: \"kubernetes.io/projected/3316930a-c6d8-48fa-8768-4638b2291be5-kube-api-access-jzjsh\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.186783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.186754    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-oauth-serving-cert\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.191590 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.191562    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9b68d5bf-crfsr"]
Nov 02 01:25:20.240347 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.240306    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 01:25:20.240347 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.240314    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 01:25:20.240562 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.240484    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 01:25:20.240562 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.240490    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 01:25:20.240562 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.240550    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 01:25:20.240562 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.240554    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 01:25:20.287563 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.287524    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-oauth-config\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.287563 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.287564    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-console-config\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.287822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.287601    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3316930a-c6d8-48fa-8768-4638b2291be5-tmp\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.287822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.287644    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-trusted-ca-bundle\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.287822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.287673    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjsh\" (UniqueName: \"kubernetes.io/projected/3316930a-c6d8-48fa-8768-4638b2291be5-kube-api-access-jzjsh\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.287822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.287705    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-oauth-serving-cert\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.287822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.287736    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-serving-cert\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.287822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.287770    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-service-ca\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.288434 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.288410    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-service-ca\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.289597 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.289572    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-oauth-serving-cert\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.290251 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.290213    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-console-config\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.290526 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.290501    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3316930a-c6d8-48fa-8768-4638b2291be5-tmp\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.290849 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.290826    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 01:25:20.292135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.292105    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-oauth-config\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.292256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.292240    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lnsht\""
Nov 02 01:25:20.302525 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.302500    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-serving-cert\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.346940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.346899    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 01:25:20.350673 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.350591    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-trusted-ca-bundle\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.354182 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.354154    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjsh\" (UniqueName: \"kubernetes.io/projected/3316930a-c6d8-48fa-8768-4638b2291be5-kube-api-access-jzjsh\") pod \"console-6d9b68d5bf-crfsr\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") " pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.458008 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.457938    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:20.910091 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:20.910071    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9b68d5bf-crfsr"]
Nov 02 01:25:20.912469 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:25:20.912444    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3316930a_c6d8_48fa_8768_4638b2291be5.slice/crio-95bc4e2b6d8352e79add3316d2e897c42d9291827446ef0536bbcce24c8d524e WatchSource:0}: Error finding container 95bc4e2b6d8352e79add3316d2e897c42d9291827446ef0536bbcce24c8d524e: Status 404 returned error can't find the container with id 95bc4e2b6d8352e79add3316d2e897c42d9291827446ef0536bbcce24c8d524e
Nov 02 01:25:21.373240 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:21.373199    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7" event={"ID":"8e58fc7d-197a-4856-8f1c-e0787c11ab2c","Type":"ContainerStarted","Data":"1576bd41d4e4aacce2ac4e2d7bc9911f822aa06b115490687dc7061c2b54e0c7"}
Nov 02 01:25:21.373428 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:21.373395    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:21.374542 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:21.374520    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9b68d5bf-crfsr" event={"ID":"3316930a-c6d8-48fa-8768-4638b2291be5","Type":"ContainerStarted","Data":"95bc4e2b6d8352e79add3316d2e897c42d9291827446ef0536bbcce24c8d524e"}
Nov 02 01:25:21.379020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:21.378999    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7"
Nov 02 01:25:21.488798 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:21.488740    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p9wf7" podStartSLOduration=1.802798846 podStartE2EDuration="3.488724222s" totalImagesPullingTime="1.685925376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:21.487824721 +0000 UTC m=+29.361465795" watchObservedRunningTime="2026-11-02 01:25:21.488724222 +0000 UTC m=+29.362365274"
Nov 02 01:25:22.522340 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.522286    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:25:22.543638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.543590    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604343    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604399    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604477    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604739 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604540    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604739 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604628    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604739 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604687    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604739 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604714    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604792    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604815    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-web-config\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604834    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.604887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604874    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605009 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.604981    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605042 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.605017    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.605045    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config-out\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.605062    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcsm\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-kube-api-access-fjcsm\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.605094    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.605108    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.605137    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.605169    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.605260 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.605194    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.700989 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.700952    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 01:25:22.701170 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.701095    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 01:25:22.701170 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.701141    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-845o6d0btpsps\""
Nov 02 01:25:22.701295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.701207    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 01:25:22.701295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.701232    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 01:25:22.701295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.701248    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 01:25:22.701295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.701262    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 01:25:22.702453 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.702351    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:25:22.705750 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705726    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705849 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705763    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705849 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705794    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705849 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705824    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705852    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705878    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705908    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705930    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705952    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.705994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.705976    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706003    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706035    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706056    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-web-config\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706079    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706101    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706133    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706158    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706182    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config-out\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706202    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcsm\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-kube-api-access-fjcsm\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.706345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.706253    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.709470 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.707594    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.709470 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.709415    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.709805 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.709777    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.710392 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.710216    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config-out\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.710392 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.710292    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.710392 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.710347    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-web-config\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.711128 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.711105    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.711292 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.711151    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.711364 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.711317    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.712104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.712084    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.749801 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.749665    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 01:25:22.749801 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.749719    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bt7w8\""
Nov 02 01:25:22.749801 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.749760    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 01:25:22.750046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.749915    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 01:25:22.750046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.749970    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 01:25:22.750046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.749985    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 01:25:22.750046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.749977    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 01:25:22.750046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.749970    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 01:25:22.750046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.750022    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 01:25:22.750221 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.750110    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 01:25:22.757485 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.757431    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.757595 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.757577    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.758151 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.758131    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.758624 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.758583    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.759841 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.759819    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.759929 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.759841    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.761131 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.761108    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.761213 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.761154    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.761344 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.761316    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.768212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.768190    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcsm\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-kube-api-access-fjcsm\") pod \"prometheus-k8s-0\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:22.953388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:22.953294    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:23.470252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:23.470209    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:25:23.475710 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:25:23.475678    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adfa7df_ca1a_40ea_81a6_dc786c77aa0e.slice/crio-e07068076e4a87ab2d8d5339958dfda0ce21cba6d705b14edec9d8b77cec78df WatchSource:0}: Error finding container e07068076e4a87ab2d8d5339958dfda0ce21cba6d705b14edec9d8b77cec78df: Status 404 returned error can't find the container with id e07068076e4a87ab2d8d5339958dfda0ce21cba6d705b14edec9d8b77cec78df
Nov 02 01:25:24.386995 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:24.386958    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerStarted","Data":"e07068076e4a87ab2d8d5339958dfda0ce21cba6d705b14edec9d8b77cec78df"}
Nov 02 01:25:25.392394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:25.392312    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9b68d5bf-crfsr" event={"ID":"3316930a-c6d8-48fa-8768-4638b2291be5","Type":"ContainerStarted","Data":"ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc"}
Nov 02 01:25:25.394237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:25.394206    4287 generic.go:356] "Generic (PLEG): container finished" podID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92" exitCode=0
Nov 02 01:25:25.394377 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:25.394285    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerDied","Data":"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"}
Nov 02 01:25:25.614339 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:25.614275    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6d9b68d5bf-crfsr" podStartSLOduration=1.709073702 podStartE2EDuration="5.614262487s" totalImagesPullingTime="3.905188785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:25.613895206 +0000 UTC m=+33.487536292" watchObservedRunningTime="2026-11-02 01:25:25.614262487 +0000 UTC m=+33.487903538"
Nov 02 01:25:28.529896 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:28.529862    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wjrcp"
Nov 02 01:25:29.415807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:29.415727    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerStarted","Data":"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"}
Nov 02 01:25:29.415807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:29.415789    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerStarted","Data":"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"}
Nov 02 01:25:30.458220 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:30.458182    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:30.458694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:30.458284    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:30.464360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:30.464335    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:31.430981 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:31.430698    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:25:32.429735 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:32.429695    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerStarted","Data":"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"}
Nov 02 01:25:32.430114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:32.429811    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerStarted","Data":"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"}
Nov 02 01:25:32.430114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:32.429852    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerStarted","Data":"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"}
Nov 02 01:25:32.430114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:32.429893    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerStarted","Data":"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"}
Nov 02 01:25:32.699883 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:32.699777    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.182969383 podStartE2EDuration="10.699762318s" totalImagesPullingTime="7.499465775s" totalInitContainerRuntime="17.32716ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:32.649945003 +0000 UTC m=+40.523586055" watchObservedRunningTime="2026-11-02 01:25:32.699762318 +0000 UTC m=+40.573403369"
Nov 02 01:25:32.954633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:25:32.954522    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:25:46.036966 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:25:46.036926    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:26:02.590176 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:02.590141    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:02.658770 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:02.658740    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:02.970290 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:02.970225    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:16.038361 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:16.038320    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:26:25.019327 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.019285    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:26:25.019878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.019769    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="prometheus" containerID="cri-o://e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809" gracePeriod=600
Nov 02 01:26:25.019878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.019778    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy-web" containerID="cri-o://bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938" gracePeriod=600
Nov 02 01:26:25.019878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.019779    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy" containerID="cri-o://acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7" gracePeriod=600
Nov 02 01:26:25.019878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.019778    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="thanos-sidecar" containerID="cri-o://aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719" gracePeriod=600
Nov 02 01:26:25.019878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.019841    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="config-reloader" containerID="cri-o://7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8" gracePeriod=600
Nov 02 01:26:25.020144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.019882    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c" gracePeriod=600
Nov 02 01:26:25.267396 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.267372    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.382632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.382560    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-2\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.382632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.382594    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-tls\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.382632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.382632    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-metrics-client-ca\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.382858 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.382649    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-serving-certs-ca-bundle\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.382858 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.382668    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-metrics-client-certs\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.382858 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.382690    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-tls-assets\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-tls-assets\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.383237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383198    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-thanos-prometheus-http-client-file\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.383341 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383261    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-kubelet-serving-ca-bundle\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.383341 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383307    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-trusted-ca-bundle\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.383341 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383311    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-metrics-client-ca" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:25.383505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383346    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-kube-api-access-fjcsm\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-kube-api-access-fjcsm\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.383505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383383    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-web-config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-web-config\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.383505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383420    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-0\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.383505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383456    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config-out\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.383505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383497    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-1\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.387087 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.383769    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-serving-certs-ca-bundle" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:25.387087 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.386062    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-1" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:25.387254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.386171    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-2" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:25.387254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.386325    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-trusted-ca-bundle" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:25.387254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.386843    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-kube-api-access-fjcsm" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "kube-api-access-fjcsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:26:25.387254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.386903    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-tls-assets" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:26:25.387254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.386903    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-kubelet-serving-ca-bundle" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:25.387254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.386952    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-metrics-client-certs" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.387254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.386988    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config-out" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:26:25.387254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387071    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-db\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.387668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387307    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.387668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387348    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-kube-rbac-proxy\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.387668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387393    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.387668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387435    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.387668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387462    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-grpc-tls\") pod \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\" (UID: \"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e\") "
Nov 02 01:26:25.387668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387540    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-thanos-prometheus-http-client-file" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387738    4287 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387759    4287 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-metrics-client-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387782    4287 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387798    4287 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-metrics-client-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387813    4287 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-tls-assets\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387829    4287 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387849    4287 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387863    4287 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387874    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjcsm\" (UniqueName: \"kubernetes.io/projected/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-kube-api-access-fjcsm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387883    4287 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config-out\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387895    4287 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.387968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.387874    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-tls" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.388565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.388102    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-0" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:26:25.388772 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.388743    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-db" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:26:25.390600 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.390570    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-kube-rbac-proxy-web" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.391888 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.391855    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.392011 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.391953    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-grpc-tls" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.392337 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.392271    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-thanos-sidecar-tls" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.392779 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.392750    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-kube-rbac-proxy" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.400087 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.400061    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-web-config" pod "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" (UID: "3adfa7df-ca1a-40ea-81a6-dc786c77aa0e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:26:25.489134 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489103    4287 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-web-config\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.489134 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489134    4287 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.489336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489148    4287 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-prometheus-k8s-db\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.489336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489162    4287 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.489336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489175    4287 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-kube-rbac-proxy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.489336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489186    4287 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-config\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.489336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489197    4287 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.489336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489209    4287 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-grpc-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.489336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.489221    4287 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:26:25.593308 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593278    4287 generic.go:356] "Generic (PLEG): container finished" podID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c" exitCode=0
Nov 02 01:26:25.593308 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593301    4287 generic.go:356] "Generic (PLEG): container finished" podID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7" exitCode=0
Nov 02 01:26:25.593308 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593307    4287 generic.go:356] "Generic (PLEG): container finished" podID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938" exitCode=0
Nov 02 01:26:25.593308 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593313    4287 generic.go:356] "Generic (PLEG): container finished" podID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719" exitCode=0
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593319    4287 generic.go:356] "Generic (PLEG): container finished" podID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8" exitCode=0
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593324    4287 generic.go:356] "Generic (PLEG): container finished" podID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809" exitCode=0
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593344    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerDied","Data":"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"}
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593377    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593427    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerDied","Data":"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"}
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593453    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerDied","Data":"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"}
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593477    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerDied","Data":"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"}
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593500    4287 scope.go:122] "RemoveContainer" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593508    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerDied","Data":"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"}
Nov 02 01:26:25.593536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593537    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerDied","Data":"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"}
Nov 02 01:26:25.593910 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.593560    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3adfa7df-ca1a-40ea-81a6-dc786c77aa0e","Type":"ContainerDied","Data":"e07068076e4a87ab2d8d5339958dfda0ce21cba6d705b14edec9d8b77cec78df"}
Nov 02 01:26:25.601404 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.601362    4287 scope.go:122] "RemoveContainer" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"
Nov 02 01:26:25.607993 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.607974    4287 scope.go:122] "RemoveContainer" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"
Nov 02 01:26:25.614555 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.614541    4287 scope.go:122] "RemoveContainer" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"
Nov 02 01:26:25.620725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.620712    4287 scope.go:122] "RemoveContainer" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"
Nov 02 01:26:25.626704 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.626679    4287 scope.go:122] "RemoveContainer" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"
Nov 02 01:26:25.633101 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.633086    4287 scope.go:122] "RemoveContainer" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"
Nov 02 01:26:25.638992 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.638978    4287 scope.go:122] "RemoveContainer" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"
Nov 02 01:26:25.639267 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:25.639248    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": container with ID starting with 13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c not found: ID does not exist" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"
Nov 02 01:26:25.639330 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.639274    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"} err="failed to get container status \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": rpc error: code = NotFound desc = could not find container \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": container with ID starting with 13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c not found: ID does not exist"
Nov 02 01:26:25.639330 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.639304    4287 scope.go:122] "RemoveContainer" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"
Nov 02 01:26:25.639539 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:25.639522    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": container with ID starting with acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7 not found: ID does not exist" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"
Nov 02 01:26:25.639581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.639543    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"} err="failed to get container status \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": rpc error: code = NotFound desc = could not find container \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": container with ID starting with acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7 not found: ID does not exist"
Nov 02 01:26:25.639581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.639557    4287 scope.go:122] "RemoveContainer" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"
Nov 02 01:26:25.639832 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:25.639814    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": container with ID starting with bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938 not found: ID does not exist" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"
Nov 02 01:26:25.639867 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.639839    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"} err="failed to get container status \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": rpc error: code = NotFound desc = could not find container \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": container with ID starting with bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938 not found: ID does not exist"
Nov 02 01:26:25.639867 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.639860    4287 scope.go:122] "RemoveContainer" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"
Nov 02 01:26:25.640106 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:25.640090    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": container with ID starting with aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719 not found: ID does not exist" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"
Nov 02 01:26:25.640144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.640111    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"} err="failed to get container status \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": rpc error: code = NotFound desc = could not find container \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": container with ID starting with aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719 not found: ID does not exist"
Nov 02 01:26:25.640144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.640124    4287 scope.go:122] "RemoveContainer" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"
Nov 02 01:26:25.640355 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:25.640338    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": container with ID starting with 7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8 not found: ID does not exist" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"
Nov 02 01:26:25.640403 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.640359    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"} err="failed to get container status \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": rpc error: code = NotFound desc = could not find container \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": container with ID starting with 7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8 not found: ID does not exist"
Nov 02 01:26:25.640403 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.640373    4287 scope.go:122] "RemoveContainer" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"
Nov 02 01:26:25.640594 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:25.640579    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": container with ID starting with e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809 not found: ID does not exist" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"
Nov 02 01:26:25.640688 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.640597    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"} err="failed to get container status \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": rpc error: code = NotFound desc = could not find container \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": container with ID starting with e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809 not found: ID does not exist"
Nov 02 01:26:25.640688 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.640627    4287 scope.go:122] "RemoveContainer" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"
Nov 02 01:26:25.640894 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:25.640878    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": container with ID starting with 15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92 not found: ID does not exist" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"
Nov 02 01:26:25.640937 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.640899    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"} err="failed to get container status \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": rpc error: code = NotFound desc = could not find container \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": container with ID starting with 15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92 not found: ID does not exist"
Nov 02 01:26:25.640937 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.640911    4287 scope.go:122] "RemoveContainer" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"
Nov 02 01:26:25.641142 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.641125    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"} err="failed to get container status \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": rpc error: code = NotFound desc = could not find container \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": container with ID starting with 13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c not found: ID does not exist"
Nov 02 01:26:25.641142 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.641142    4287 scope.go:122] "RemoveContainer" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"
Nov 02 01:26:25.641370 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.641352    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"} err="failed to get container status \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": rpc error: code = NotFound desc = could not find container \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": container with ID starting with acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7 not found: ID does not exist"
Nov 02 01:26:25.641420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.641370    4287 scope.go:122] "RemoveContainer" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"
Nov 02 01:26:25.641631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.641591    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"} err="failed to get container status \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": rpc error: code = NotFound desc = could not find container \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": container with ID starting with bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938 not found: ID does not exist"
Nov 02 01:26:25.641704 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.641632    4287 scope.go:122] "RemoveContainer" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"
Nov 02 01:26:25.641879 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.641857    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"} err="failed to get container status \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": rpc error: code = NotFound desc = could not find container \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": container with ID starting with aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719 not found: ID does not exist"
Nov 02 01:26:25.641949 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.641881    4287 scope.go:122] "RemoveContainer" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"
Nov 02 01:26:25.642121 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642103    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"} err="failed to get container status \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": rpc error: code = NotFound desc = could not find container \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": container with ID starting with 7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8 not found: ID does not exist"
Nov 02 01:26:25.642165 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642121    4287 scope.go:122] "RemoveContainer" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"
Nov 02 01:26:25.642337 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642318    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"} err="failed to get container status \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": rpc error: code = NotFound desc = could not find container \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": container with ID starting with e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809 not found: ID does not exist"
Nov 02 01:26:25.642413 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642338    4287 scope.go:122] "RemoveContainer" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"
Nov 02 01:26:25.642556 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642538    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"} err="failed to get container status \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": rpc error: code = NotFound desc = could not find container \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": container with ID starting with 15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92 not found: ID does not exist"
Nov 02 01:26:25.642601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642556    4287 scope.go:122] "RemoveContainer" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"
Nov 02 01:26:25.642775 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642758    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"} err="failed to get container status \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": rpc error: code = NotFound desc = could not find container \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": container with ID starting with 13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c not found: ID does not exist"
Nov 02 01:26:25.642841 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642776    4287 scope.go:122] "RemoveContainer" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"
Nov 02 01:26:25.642996 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642977    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"} err="failed to get container status \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": rpc error: code = NotFound desc = could not find container \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": container with ID starting with acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7 not found: ID does not exist"
Nov 02 01:26:25.643043 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.642996    4287 scope.go:122] "RemoveContainer" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"
Nov 02 01:26:25.643182 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643163    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"} err="failed to get container status \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": rpc error: code = NotFound desc = could not find container \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": container with ID starting with bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938 not found: ID does not exist"
Nov 02 01:26:25.643227 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643183    4287 scope.go:122] "RemoveContainer" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"
Nov 02 01:26:25.643365 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643343    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"} err="failed to get container status \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": rpc error: code = NotFound desc = could not find container \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": container with ID starting with aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719 not found: ID does not exist"
Nov 02 01:26:25.643430 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643365    4287 scope.go:122] "RemoveContainer" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"
Nov 02 01:26:25.643561 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643542    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"} err="failed to get container status \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": rpc error: code = NotFound desc = could not find container \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": container with ID starting with 7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8 not found: ID does not exist"
Nov 02 01:26:25.643561 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643559    4287 scope.go:122] "RemoveContainer" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"
Nov 02 01:26:25.643783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643763    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"} err="failed to get container status \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": rpc error: code = NotFound desc = could not find container \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": container with ID starting with e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809 not found: ID does not exist"
Nov 02 01:26:25.643831 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643783    4287 scope.go:122] "RemoveContainer" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"
Nov 02 01:26:25.643978 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643959    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"} err="failed to get container status \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": rpc error: code = NotFound desc = could not find container \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": container with ID starting with 15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92 not found: ID does not exist"
Nov 02 01:26:25.644044 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.643980    4287 scope.go:122] "RemoveContainer" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"
Nov 02 01:26:25.644190 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644170    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"} err="failed to get container status \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": rpc error: code = NotFound desc = could not find container \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": container with ID starting with 13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c not found: ID does not exist"
Nov 02 01:26:25.644235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644191    4287 scope.go:122] "RemoveContainer" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"
Nov 02 01:26:25.644383 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644368    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"} err="failed to get container status \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": rpc error: code = NotFound desc = could not find container \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": container with ID starting with acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7 not found: ID does not exist"
Nov 02 01:26:25.644383 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644382    4287 scope.go:122] "RemoveContainer" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"
Nov 02 01:26:25.644556 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644541    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"} err="failed to get container status \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": rpc error: code = NotFound desc = could not find container \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": container with ID starting with bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938 not found: ID does not exist"
Nov 02 01:26:25.644595 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644557    4287 scope.go:122] "RemoveContainer" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"
Nov 02 01:26:25.644785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644767    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"} err="failed to get container status \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": rpc error: code = NotFound desc = could not find container \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": container with ID starting with aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719 not found: ID does not exist"
Nov 02 01:26:25.644834 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644786    4287 scope.go:122] "RemoveContainer" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"
Nov 02 01:26:25.644975 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644958    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"} err="failed to get container status \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": rpc error: code = NotFound desc = could not find container \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": container with ID starting with 7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8 not found: ID does not exist"
Nov 02 01:26:25.645041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.644976    4287 scope.go:122] "RemoveContainer" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"
Nov 02 01:26:25.645152 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645137    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"} err="failed to get container status \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": rpc error: code = NotFound desc = could not find container \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": container with ID starting with e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809 not found: ID does not exist"
Nov 02 01:26:25.645193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645152    4287 scope.go:122] "RemoveContainer" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"
Nov 02 01:26:25.645316 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645301    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"} err="failed to get container status \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": rpc error: code = NotFound desc = could not find container \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": container with ID starting with 15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92 not found: ID does not exist"
Nov 02 01:26:25.645358 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645316    4287 scope.go:122] "RemoveContainer" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"
Nov 02 01:26:25.645485 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645468    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"} err="failed to get container status \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": rpc error: code = NotFound desc = could not find container \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": container with ID starting with 13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c not found: ID does not exist"
Nov 02 01:26:25.645550 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645486    4287 scope.go:122] "RemoveContainer" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"
Nov 02 01:26:25.645691 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645675    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"} err="failed to get container status \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": rpc error: code = NotFound desc = could not find container \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": container with ID starting with acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7 not found: ID does not exist"
Nov 02 01:26:25.645740 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645690    4287 scope.go:122] "RemoveContainer" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"
Nov 02 01:26:25.645920 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645905    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"} err="failed to get container status \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": rpc error: code = NotFound desc = could not find container \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": container with ID starting with bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938 not found: ID does not exist"
Nov 02 01:26:25.645975 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.645921    4287 scope.go:122] "RemoveContainer" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"
Nov 02 01:26:25.646130 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.646111    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"} err="failed to get container status \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": rpc error: code = NotFound desc = could not find container \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": container with ID starting with aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719 not found: ID does not exist"
Nov 02 01:26:25.646130 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.646129    4287 scope.go:122] "RemoveContainer" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"
Nov 02 01:26:25.646354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.646334    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"} err="failed to get container status \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": rpc error: code = NotFound desc = could not find container \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": container with ID starting with 7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8 not found: ID does not exist"
Nov 02 01:26:25.646426 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.646356    4287 scope.go:122] "RemoveContainer" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"
Nov 02 01:26:25.646643 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.646625    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"} err="failed to get container status \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": rpc error: code = NotFound desc = could not find container \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": container with ID starting with e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809 not found: ID does not exist"
Nov 02 01:26:25.646718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.646645    4287 scope.go:122] "RemoveContainer" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"
Nov 02 01:26:25.646887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.646865    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"} err="failed to get container status \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": rpc error: code = NotFound desc = could not find container \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": container with ID starting with 15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92 not found: ID does not exist"
Nov 02 01:26:25.646955 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.646889    4287 scope.go:122] "RemoveContainer" containerID="13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"
Nov 02 01:26:25.647124 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647106    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c"} err="failed to get container status \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": rpc error: code = NotFound desc = could not find container \"13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c\": container with ID starting with 13ebfe1ac66e33a8005a86c5b7a1ac04450240bd36b933ed74b2cc7b1b40e35c not found: ID does not exist"
Nov 02 01:26:25.647180 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647124    4287 scope.go:122] "RemoveContainer" containerID="acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"
Nov 02 01:26:25.647332 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647308    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7"} err="failed to get container status \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": rpc error: code = NotFound desc = could not find container \"acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7\": container with ID starting with acafa15a479f97010eebe84c2078d2e292446b26534069a442fcfea7c19491d7 not found: ID does not exist"
Nov 02 01:26:25.647332 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647328    4287 scope.go:122] "RemoveContainer" containerID="bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"
Nov 02 01:26:25.647530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647512    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938"} err="failed to get container status \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": rpc error: code = NotFound desc = could not find container \"bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938\": container with ID starting with bfc439b832f256b56b25f6330caee0b18424f0a4b2735346bd42acc05dfc1938 not found: ID does not exist"
Nov 02 01:26:25.647530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647528    4287 scope.go:122] "RemoveContainer" containerID="aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"
Nov 02 01:26:25.647765 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647745    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719"} err="failed to get container status \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": rpc error: code = NotFound desc = could not find container \"aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719\": container with ID starting with aa344314712b189ba176d60bd8ba5b6249c9df984b6f87fd911d413fd8eb0719 not found: ID does not exist"
Nov 02 01:26:25.647811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647765    4287 scope.go:122] "RemoveContainer" containerID="7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"
Nov 02 01:26:25.647977 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647964    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8"} err="failed to get container status \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": rpc error: code = NotFound desc = could not find container \"7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8\": container with ID starting with 7a59b67bc99f17b5ffaa3ce7a4494b7a0215d1bdeb2bad1f2746267b1650f2a8 not found: ID does not exist"
Nov 02 01:26:25.648029 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.647977    4287 scope.go:122] "RemoveContainer" containerID="e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"
Nov 02 01:26:25.648173 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.648158    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809"} err="failed to get container status \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": rpc error: code = NotFound desc = could not find container \"e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809\": container with ID starting with e6932fd44aec6864d82ecf9f13bdb1c99bb5afe81c4e899fbaa520146691b809 not found: ID does not exist"
Nov 02 01:26:25.648217 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.648174    4287 scope.go:122] "RemoveContainer" containerID="15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"
Nov 02 01:26:25.648361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.648344    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92"} err="failed to get container status \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": rpc error: code = NotFound desc = could not find container \"15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92\": container with ID starting with 15bdddb29ec2ee2784c893e1edf7eb70f13deb6cd366faa2d6ed7242a08b2d92 not found: ID does not exist"
Nov 02 01:26:25.762527 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.762500    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:26:25.770549 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.770528    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:26:25.803249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803228    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:26:25.803465 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803453    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="init-config-reloader"
Nov 02 01:26:25.803503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803467    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="init-config-reloader"
Nov 02 01:26:25.803503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803475    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="prometheus"
Nov 02 01:26:25.803503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803480    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="prometheus"
Nov 02 01:26:25.803503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803489    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="thanos-sidecar"
Nov 02 01:26:25.803503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803494    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="thanos-sidecar"
Nov 02 01:26:25.803503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803500    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy-web"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803505    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy-web"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803513    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="config-reloader"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803518    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="config-reloader"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803522    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy-thanos"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803527    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy-thanos"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803536    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803540    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803576    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="prometheus"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803583    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="thanos-sidecar"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803588    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy-web"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803594    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="config-reloader"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803600    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy-thanos"
Nov 02 01:26:25.803687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.803635    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" containerName="kube-rbac-proxy"
Nov 02 01:26:25.807133 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.807116    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.862472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.862241    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 01:26:25.862472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.862312    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 01:26:25.862472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.862351    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 01:26:25.862472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.862411    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 01:26:25.891579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891518    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.891579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891555    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.891720 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891582    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.891751 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891725    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.891789 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891751    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.891789 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891767    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.891892 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891811    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.891892 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891855    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891907    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891963    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.891991    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892152 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892024    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892152 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892054    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892152 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892081    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6aceefcc-71d6-40e5-bac0-bfed1932116a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892260 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892151    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-config\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892260 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892200    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892260 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892230    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892260 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892256    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6aceefcc-71d6-40e5-bac0-bfed1932116a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892436 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892281    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.892436 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.892340    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfqm\" (UniqueName: \"kubernetes.io/projected/6aceefcc-71d6-40e5-bac0-bfed1932116a-kube-api-access-2gfqm\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.911533 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.911508    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 01:26:25.912573 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912548    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 01:26:25.912741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912589    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 01:26:25.912741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912602    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 01:26:25.912741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912633    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 01:26:25.912741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912673    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 01:26:25.912741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912695    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-845o6d0btpsps\""
Nov 02 01:26:25.912741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912744    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 01:26:25.913065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912752    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 01:26:25.913065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912830    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 01:26:25.913065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912882    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bt7w8\""
Nov 02 01:26:25.913065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912891    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 01:26:25.913065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.912935    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 01:26:25.934744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.934719    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:26:25.993593 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993568    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-config\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.993721 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993599    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.993721 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993656    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.993721 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993687    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6aceefcc-71d6-40e5-bac0-bfed1932116a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.993721 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993711    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.993951 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993736    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfqm\" (UniqueName: \"kubernetes.io/projected/6aceefcc-71d6-40e5-bac0-bfed1932116a-kube-api-access-2gfqm\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.993951 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993870    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.993951 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993911    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.993951 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993939    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993968    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.993996    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994027    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994052    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994076    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994121    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994163    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994187    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994212    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994241    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994269    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6aceefcc-71d6-40e5-bac0-bfed1932116a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.994435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994425    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.996593 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.994747    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.996593 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.995388    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.996593 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.995933    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.996593 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.996003    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.996593 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.996551    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.996970 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.996946    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.997162 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.997134    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.997237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.997212    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6aceefcc-71d6-40e5-bac0-bfed1932116a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.998058 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.997514    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-config\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.998058 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.998015    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.998701 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.998673    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6aceefcc-71d6-40e5-bac0-bfed1932116a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.998976 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.998895    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.999564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.999106    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6aceefcc-71d6-40e5-bac0-bfed1932116a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.999564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.999224    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.999564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.999473    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.999564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.999494    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:25.999814 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:25.999589    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:26.000570 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:26.000548    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aceefcc-71d6-40e5-bac0-bfed1932116a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:26.055149 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:26.055124    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfqm\" (UniqueName: \"kubernetes.io/projected/6aceefcc-71d6-40e5-bac0-bfed1932116a-kube-api-access-2gfqm\") pod \"prometheus-k8s-0\" (UID: \"6aceefcc-71d6-40e5-bac0-bfed1932116a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:26.123097 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:26.123065    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:26.235908 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:26.235877    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3adfa7df-ca1a-40ea-81a6-dc786c77aa0e" path="/var/lib/kubelet/pods/3adfa7df-ca1a-40ea-81a6-dc786c77aa0e/volumes"
Nov 02 01:26:26.551274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:26.551246    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 01:26:26.554207 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:26:26.554182    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aceefcc_71d6_40e5_bac0_bfed1932116a.slice/crio-f5a74b8a9d5581d1495c993f7659efb76e7ec84001cacc1d1ac79c2e1f11dc68 WatchSource:0}: Error finding container f5a74b8a9d5581d1495c993f7659efb76e7ec84001cacc1d1ac79c2e1f11dc68: Status 404 returned error can't find the container with id f5a74b8a9d5581d1495c993f7659efb76e7ec84001cacc1d1ac79c2e1f11dc68
Nov 02 01:26:26.598638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:26.598585    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6aceefcc-71d6-40e5-bac0-bfed1932116a","Type":"ContainerStarted","Data":"f5a74b8a9d5581d1495c993f7659efb76e7ec84001cacc1d1ac79c2e1f11dc68"}
Nov 02 01:26:27.605256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:27.605227    4287 generic.go:356] "Generic (PLEG): container finished" podID="6aceefcc-71d6-40e5-bac0-bfed1932116a" containerID="52d3752d7564334e186527b5b70d2dd3f921081ef6b8da909cd3bed84c43f2f1" exitCode=0
Nov 02 01:26:27.605658 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:27.605311    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6aceefcc-71d6-40e5-bac0-bfed1932116a","Type":"ContainerDied","Data":"52d3752d7564334e186527b5b70d2dd3f921081ef6b8da909cd3bed84c43f2f1"}
Nov 02 01:26:28.613121 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:28.613083    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6aceefcc-71d6-40e5-bac0-bfed1932116a","Type":"ContainerStarted","Data":"d8026fcd5659b2b8ebd3a0b4189726bbfd06920e79845aefa5000a402189fc3e"}
Nov 02 01:26:28.613601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:28.613158    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6aceefcc-71d6-40e5-bac0-bfed1932116a","Type":"ContainerStarted","Data":"bcad6a0f83699c50dd5acbf0bd680ec9ffb56491f68674fad797274816a6366f"}
Nov 02 01:26:28.613601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:28.613203    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6aceefcc-71d6-40e5-bac0-bfed1932116a","Type":"ContainerStarted","Data":"3f73a24a1e8c5b7c3cc88d9faab5bb96784d1134c0a1cae74df0bbd20dc7c388"}
Nov 02 01:26:28.613601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:28.613247    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6aceefcc-71d6-40e5-bac0-bfed1932116a","Type":"ContainerStarted","Data":"f32d186381d44fdf38123671c93e9e2aaa345c637853d0d60035e32d96edf48f"}
Nov 02 01:26:28.613601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:28.613294    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6aceefcc-71d6-40e5-bac0-bfed1932116a","Type":"ContainerStarted","Data":"3e3d5130b6823ed3f4bbe64c0c4932b718681f7c59eb5ad304969ffa36278dfd"}
Nov 02 01:26:28.613601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:28.613335    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6aceefcc-71d6-40e5-bac0-bfed1932116a","Type":"ContainerStarted","Data":"fb3f7d4ef9c324cd7af94ed14243e4e9121d87bb6d4f97396346ac1767deb304"}
Nov 02 01:26:28.745528 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:28.745474    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.729569396 podStartE2EDuration="3.745454773s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.885377ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:26:28.744976619 +0000 UTC m=+96.618617672" watchObservedRunningTime="2026-11-02 01:26:28.745454773 +0000 UTC m=+96.619095821"
Nov 02 01:26:31.123639 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:31.123579    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:26:37.747305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:26:37.747269    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9b68d5bf-crfsr"]
Nov 02 01:26:46.036929 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:26:46.036894    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:27:02.766725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:02.766667    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6d9b68d5bf-crfsr" podUID="3316930a-c6d8-48fa-8768-4638b2291be5" containerName="console" containerID="cri-o://ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc" gracePeriod=15
Nov 02 01:27:03.007941 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.007915    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:27:03.163850 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.163770    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-service-ca\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-service-ca\") pod \"3316930a-c6d8-48fa-8768-4638b2291be5\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") "
Nov 02 01:27:03.163850 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.163810    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3316930a-c6d8-48fa-8768-4638b2291be5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3316930a-c6d8-48fa-8768-4638b2291be5-tmp\") pod \"3316930a-c6d8-48fa-8768-4638b2291be5\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") "
Nov 02 01:27:03.163850 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.163840    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-trusted-ca-bundle\") pod \"3316930a-c6d8-48fa-8768-4638b2291be5\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") "
Nov 02 01:27:03.164143 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.163857    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-oauth-serving-cert\") pod \"3316930a-c6d8-48fa-8768-4638b2291be5\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") "
Nov 02 01:27:03.164143 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.163997    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3316930a-c6d8-48fa-8768-4638b2291be5-kube-api-access-jzjsh\" (UniqueName: \"kubernetes.io/projected/3316930a-c6d8-48fa-8768-4638b2291be5-kube-api-access-jzjsh\") pod \"3316930a-c6d8-48fa-8768-4638b2291be5\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") "
Nov 02 01:27:03.164143 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164046    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-console-config\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-console-config\") pod \"3316930a-c6d8-48fa-8768-4638b2291be5\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") "
Nov 02 01:27:03.164143 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164103    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-serving-cert\") pod \"3316930a-c6d8-48fa-8768-4638b2291be5\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") "
Nov 02 01:27:03.164143 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164110    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3316930a-c6d8-48fa-8768-4638b2291be5-tmp" pod "3316930a-c6d8-48fa-8768-4638b2291be5" (UID: "3316930a-c6d8-48fa-8768-4638b2291be5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:27:03.164393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164137    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-oauth-config\") pod \"3316930a-c6d8-48fa-8768-4638b2291be5\" (UID: \"3316930a-c6d8-48fa-8768-4638b2291be5\") "
Nov 02 01:27:03.164393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164233    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-service-ca" pod "3316930a-c6d8-48fa-8768-4638b2291be5" (UID: "3316930a-c6d8-48fa-8768-4638b2291be5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:03.164393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164248    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-trusted-ca-bundle" pod "3316930a-c6d8-48fa-8768-4638b2291be5" (UID: "3316930a-c6d8-48fa-8768-4638b2291be5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:03.164393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164263    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-oauth-serving-cert" pod "3316930a-c6d8-48fa-8768-4638b2291be5" (UID: "3316930a-c6d8-48fa-8768-4638b2291be5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:03.164679 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164450    4287 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:27:03.164679 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164468    4287 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3316930a-c6d8-48fa-8768-4638b2291be5-tmp\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:27:03.164679 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164483    4287 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-trusted-ca-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:27:03.164679 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164496    4287 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-oauth-serving-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:27:03.164827 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.164677    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-console-config" pod "3316930a-c6d8-48fa-8768-4638b2291be5" (UID: "3316930a-c6d8-48fa-8768-4638b2291be5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:27:03.166439 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.166407    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3316930a-c6d8-48fa-8768-4638b2291be5-kube-api-access-jzjsh" pod "3316930a-c6d8-48fa-8768-4638b2291be5" (UID: "3316930a-c6d8-48fa-8768-4638b2291be5"). InnerVolumeSpecName "kube-api-access-jzjsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:27:03.166530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.166439    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-oauth-config" pod "3316930a-c6d8-48fa-8768-4638b2291be5" (UID: "3316930a-c6d8-48fa-8768-4638b2291be5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:03.166530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.166455    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-serving-cert" pod "3316930a-c6d8-48fa-8768-4638b2291be5" (UID: "3316930a-c6d8-48fa-8768-4638b2291be5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:27:03.265323 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.265295    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjsh\" (UniqueName: \"kubernetes.io/projected/3316930a-c6d8-48fa-8768-4638b2291be5-kube-api-access-jzjsh\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:27:03.265323 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.265321    4287 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3316930a-c6d8-48fa-8768-4638b2291be5-console-config\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:27:03.265503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.265332    4287 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-serving-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:27:03.265503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.265341    4287 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3316930a-c6d8-48fa-8768-4638b2291be5-console-oauth-config\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:27:03.728362 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.728329    4287 generic.go:356] "Generic (PLEG): container finished" podID="3316930a-c6d8-48fa-8768-4638b2291be5" containerID="ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc" exitCode=0
Nov 02 01:27:03.728517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.728369    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9b68d5bf-crfsr"
Nov 02 01:27:03.728517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.728407    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9b68d5bf-crfsr" event={"ID":"3316930a-c6d8-48fa-8768-4638b2291be5","Type":"ContainerDied","Data":"ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc"}
Nov 02 01:27:03.728517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.728457    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9b68d5bf-crfsr" event={"ID":"3316930a-c6d8-48fa-8768-4638b2291be5","Type":"ContainerDied","Data":"95bc4e2b6d8352e79add3316d2e897c42d9291827446ef0536bbcce24c8d524e"}
Nov 02 01:27:03.728517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.728479    4287 scope.go:122] "RemoveContainer" containerID="ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc"
Nov 02 01:27:03.736235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.736219    4287 scope.go:122] "RemoveContainer" containerID="ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc"
Nov 02 01:27:03.736498 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:27:03.736482    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc\": container with ID starting with ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc not found: ID does not exist" containerID="ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc"
Nov 02 01:27:03.736555 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.736504    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc"} err="failed to get container status \"ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc\": rpc error: code = NotFound desc = could not find container \"ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc\": container with ID starting with ce42fde1a0a1a01f703a8fe0e08f7bfbddcb79abd6b908baa4aa84b5617061fc not found: ID does not exist"
Nov 02 01:27:03.891768 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.891737    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9b68d5bf-crfsr"]
Nov 02 01:27:03.941253 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:03.941223    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d9b68d5bf-crfsr"]
Nov 02 01:27:04.235730 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:04.235701    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3316930a-c6d8-48fa-8768-4638b2291be5" path="/var/lib/kubelet/pods/3316930a-c6d8-48fa-8768-4638b2291be5/volumes"
Nov 02 01:27:16.038229 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:27:16.038194    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:27:26.123935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:26.123896    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:27:26.139546 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:26.139516    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:27:26.814321 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:27:26.814296    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 01:27:46.036899 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:27:46.036860    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:28:16.038567 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:28:16.038531    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:28:46.037168 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:28:46.037132    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:29:16.037060 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:29:16.037012    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:29:46.036959 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:29:46.036924    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:29:52.232402 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:29:52.232367    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:29:52.232862 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:29:52.232776    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:30:14.705356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.702918    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"]
Nov 02 01:30:14.705356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.703443    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3316930a-c6d8-48fa-8768-4638b2291be5" containerName="console"
Nov 02 01:30:14.705356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.703460    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3316930a-c6d8-48fa-8768-4638b2291be5" containerName="console"
Nov 02 01:30:14.705356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.703652    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3316930a-c6d8-48fa-8768-4638b2291be5" containerName="console"
Nov 02 01:30:14.707469 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.707450    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.741749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.741714    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.741749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.741752    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.741930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.741778    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgjnz\" (UniqueName: \"kubernetes.io/projected/2909310a-fff5-4192-afba-1f75800db215-kube-api-access-pgjnz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.811930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.811889    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 01:30:14.811930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.811896    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 01:30:14.812123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.811983    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hg9l6\""
Nov 02 01:30:14.812877 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.812856    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"]
Nov 02 01:30:14.842256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.842222    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.842414 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.842265    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.842414 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.842292    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgjnz\" (UniqueName: \"kubernetes.io/projected/2909310a-fff5-4192-afba-1f75800db215-kube-api-access-pgjnz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.842673 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.842653    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.842743 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.842670    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:14.903292 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:14.903256    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgjnz\" (UniqueName: \"kubernetes.io/projected/2909310a-fff5-4192-afba-1f75800db215-kube-api-access-pgjnz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:15.022548 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:15.022513    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:15.397055 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:15.396987    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"]
Nov 02 01:30:15.400348 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:15.400318    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2909310a_fff5_4192_afba_1f75800db215.slice/crio-2b8c9cce6ab69b475f2640af0b117d3e15089a5155f8779a396ca50418b067be WatchSource:0}: Error finding container 2b8c9cce6ab69b475f2640af0b117d3e15089a5155f8779a396ca50418b067be: Status 404 returned error can't find the container with id 2b8c9cce6ab69b475f2640af0b117d3e15089a5155f8779a396ca50418b067be
Nov 02 01:30:15.402622 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:15.402594    4287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:30:16.036926 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:30:16.036893    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:30:16.333919 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.333832    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl" event={"ID":"2909310a-fff5-4192-afba-1f75800db215","Type":"ContainerStarted","Data":"2b8c9cce6ab69b475f2640af0b117d3e15089a5155f8779a396ca50418b067be"}
Nov 02 01:30:16.709832 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.709751    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"]
Nov 02 01:30:16.713262 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.713233    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.755072 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.755041    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.755249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.755084    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.755249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.755228    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td5v8\" (UniqueName: \"kubernetes.io/projected/ec56b813-e200-4cd9-9d9a-0bded6d7644d-kube-api-access-td5v8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.815056 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.815021    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-wzxmv\""
Nov 02 01:30:16.816866 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.816842    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"]
Nov 02 01:30:16.817265 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.817246    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"]
Nov 02 01:30:16.820494 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.820476    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.856201 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.856162    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.856438 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.856211    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td5v8\" (UniqueName: \"kubernetes.io/projected/ec56b813-e200-4cd9-9d9a-0bded6d7644d-kube-api-access-td5v8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.856438 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.856305    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4kc\" (UniqueName: \"kubernetes.io/projected/be5e982b-e76a-454d-a38f-ac77343074a9-kube-api-access-6x4kc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.856438 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.856357    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.856438 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.856395    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.856725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.856459    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.856784 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.856765    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.857186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.856839    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.873223 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.873139    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"]
Nov 02 01:30:16.874241 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.874207    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-kvjzn\""
Nov 02 01:30:16.917504 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.917472    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"]
Nov 02 01:30:16.917872 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.917852    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td5v8\" (UniqueName: \"kubernetes.io/projected/ec56b813-e200-4cd9-9d9a-0bded6d7644d-kube-api-access-td5v8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:16.920977 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.920954    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:16.957106 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.957078    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.957269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.957120    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:16.957269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.957146    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:16.957269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.957247    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.957447 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.957309    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4kc\" (UniqueName: \"kubernetes.io/projected/be5e982b-e76a-454d-a38f-ac77343074a9-kube-api-access-6x4kc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.957447 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.957340    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scf45\" (UniqueName: \"kubernetes.io/projected/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-kube-api-access-scf45\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:16.957546 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.957496    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.957639 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.957621    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:16.974478 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.974401    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-v4znv\""
Nov 02 01:30:16.979278 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:16.979251    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"]
Nov 02 01:30:17.013098 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.013068    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"]
Nov 02 01:30:17.016497 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.016470    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.021784 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.021752    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4kc\" (UniqueName: \"kubernetes.io/projected/be5e982b-e76a-454d-a38f-ac77343074a9-kube-api-access-6x4kc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:17.031477 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.031380    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:17.058930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.058900    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scf45\" (UniqueName: \"kubernetes.io/projected/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-kube-api-access-scf45\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:17.059304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.058952    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.059304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.059004    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.059304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.059038    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-622s7\" (UniqueName: \"kubernetes.io/projected/5cf780e8-26b5-4778-a680-8c08f2dca6a7-kube-api-access-622s7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.059304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.059190    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:17.059304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.059245    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:17.059806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.059708    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:17.059806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.059703    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:17.075485 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.075148    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"]
Nov 02 01:30:17.075485 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.075433    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-q6lxd\""
Nov 02 01:30:17.128630 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.124866    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scf45\" (UniqueName: \"kubernetes.io/projected/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-kube-api-access-scf45\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:17.139530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.139498    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:17.160401 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.160354    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.160570 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.160430    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.160570 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.160470    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-622s7\" (UniqueName: \"kubernetes.io/projected/5cf780e8-26b5-4778-a680-8c08f2dca6a7-kube-api-access-622s7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.161167 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.161116    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.161262 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.161176    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.222540 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.222509    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-622s7\" (UniqueName: \"kubernetes.io/projected/5cf780e8-26b5-4778-a680-8c08f2dca6a7-kube-api-access-622s7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.241280 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.241203    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:17.341620 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.341575    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:17.414824 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.414713    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"]
Nov 02 01:30:17.419713 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:17.419660    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec56b813_e200_4cd9_9d9a_0bded6d7644d.slice/crio-90947635ffd3c1834ad59a34e05d989e1005a4118b90b1911e7802a8f3252b4e WatchSource:0}: Error finding container 90947635ffd3c1834ad59a34e05d989e1005a4118b90b1911e7802a8f3252b4e: Status 404 returned error can't find the container with id 90947635ffd3c1834ad59a34e05d989e1005a4118b90b1911e7802a8f3252b4e
Nov 02 01:30:17.520724 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.520690    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"]
Nov 02 01:30:17.524381 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:17.524346    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5e982b_e76a_454d_a38f_ac77343074a9.slice/crio-4f8143594531d550793e0ac1d193919b5696e7829af2cd82951ccd43020765cf WatchSource:0}: Error finding container 4f8143594531d550793e0ac1d193919b5696e7829af2cd82951ccd43020765cf: Status 404 returned error can't find the container with id 4f8143594531d550793e0ac1d193919b5696e7829af2cd82951ccd43020765cf
Nov 02 01:30:17.618823 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.618664    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"]
Nov 02 01:30:17.621582 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:17.621549    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fb3dc4_1a15_4290_905f_d413cd3d98f1.slice/crio-edb8833b3a0f1e749bd1dce3e7fb742cbb1a2bbe2508a46ba5f66d27f3d8d8de WatchSource:0}: Error finding container edb8833b3a0f1e749bd1dce3e7fb742cbb1a2bbe2508a46ba5f66d27f3d8d8de: Status 404 returned error can't find the container with id edb8833b3a0f1e749bd1dce3e7fb742cbb1a2bbe2508a46ba5f66d27f3d8d8de
Nov 02 01:30:17.726057 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:17.726022    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"]
Nov 02 01:30:17.729699 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:17.729672    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf780e8_26b5_4778_a680_8c08f2dca6a7.slice/crio-94f4d61878ea1d3a957a4a42c5cdfb254277a1ba7d1525498c9834f903f97df1 WatchSource:0}: Error finding container 94f4d61878ea1d3a957a4a42c5cdfb254277a1ba7d1525498c9834f903f97df1: Status 404 returned error can't find the container with id 94f4d61878ea1d3a957a4a42c5cdfb254277a1ba7d1525498c9834f903f97df1
Nov 02 01:30:18.343275 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.343238    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp" event={"ID":"5cf780e8-26b5-4778-a680-8c08f2dca6a7","Type":"ContainerStarted","Data":"94f4d61878ea1d3a957a4a42c5cdfb254277a1ba7d1525498c9834f903f97df1"}
Nov 02 01:30:18.344494 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.344465    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g" event={"ID":"a3fb3dc4-1a15-4290-905f-d413cd3d98f1","Type":"ContainerStarted","Data":"edb8833b3a0f1e749bd1dce3e7fb742cbb1a2bbe2508a46ba5f66d27f3d8d8de"}
Nov 02 01:30:18.346015 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.345954    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb" event={"ID":"be5e982b-e76a-454d-a38f-ac77343074a9","Type":"ContainerStarted","Data":"4f8143594531d550793e0ac1d193919b5696e7829af2cd82951ccd43020765cf"}
Nov 02 01:30:18.347304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.347282    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm" event={"ID":"ec56b813-e200-4cd9-9d9a-0bded6d7644d","Type":"ContainerStarted","Data":"90947635ffd3c1834ad59a34e05d989e1005a4118b90b1911e7802a8f3252b4e"}
Nov 02 01:30:18.457430 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.457392    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"]
Nov 02 01:30:18.475861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.475826    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.519907 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.519878    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"]
Nov 02 01:30:18.528472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.528282    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-z5hsq\""
Nov 02 01:30:18.574003 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.573952    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc229\" (UniqueName: \"kubernetes.io/projected/2ae86c75-5bbf-49bd-a101-5962cd40b007-kube-api-access-lc229\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.574171 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.574020    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.574171 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.574067    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.676105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.674847    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc229\" (UniqueName: \"kubernetes.io/projected/2ae86c75-5bbf-49bd-a101-5962cd40b007-kube-api-access-lc229\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.676105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.674912    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.676105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.674973    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.676105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.675416    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.676105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.676011    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.752922 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.752886    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc229\" (UniqueName: \"kubernetes.io/projected/2ae86c75-5bbf-49bd-a101-5962cd40b007-kube-api-access-lc229\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:18.799668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:18.799339    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:19.029852 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.029815    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"]
Nov 02 01:30:19.034347 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.033812    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.079685 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.079644    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckj4\" (UniqueName: \"kubernetes.io/projected/5318f7c6-23a6-4de7-8f52-a54499d12bf9-kube-api-access-zckj4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.079866 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.079705    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.079866 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.079816    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.091264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.091226    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-5wn92\""
Nov 02 01:30:19.093871 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.093813    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"]
Nov 02 01:30:19.180457 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.180414    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckj4\" (UniqueName: \"kubernetes.io/projected/5318f7c6-23a6-4de7-8f52-a54499d12bf9-kube-api-access-zckj4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.180677 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.180477    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.180677 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.180515    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.181689 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.180982    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.181689 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.181554    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.217408 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.217369    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"]
Nov 02 01:30:19.227212 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:19.227170    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae86c75_5bbf_49bd_a101_5962cd40b007.slice/crio-2ac3fc1329754c78f4ba2a4fcf5cdafa46fdd30914b5b5cdb79a5cab8b592378 WatchSource:0}: Error finding container 2ac3fc1329754c78f4ba2a4fcf5cdafa46fdd30914b5b5cdb79a5cab8b592378: Status 404 returned error can't find the container with id 2ac3fc1329754c78f4ba2a4fcf5cdafa46fdd30914b5b5cdb79a5cab8b592378
Nov 02 01:30:19.244298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.244274    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckj4\" (UniqueName: \"kubernetes.io/projected/5318f7c6-23a6-4de7-8f52-a54499d12bf9-kube-api-access-zckj4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.354075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.354003    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:19.354794 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.354720    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v" event={"ID":"2ae86c75-5bbf-49bd-a101-5962cd40b007","Type":"ContainerStarted","Data":"2ac3fc1329754c78f4ba2a4fcf5cdafa46fdd30914b5b5cdb79a5cab8b592378"}
Nov 02 01:30:19.766737 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:19.765852    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"]
Nov 02 01:30:19.788856 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:19.788815    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5318f7c6_23a6_4de7_8f52_a54499d12bf9.slice/crio-736fac086813d87ae3885bf3059fb75304a46b48e3d90ee1ea85c26b09e000bb WatchSource:0}: Error finding container 736fac086813d87ae3885bf3059fb75304a46b48e3d90ee1ea85c26b09e000bb: Status 404 returned error can't find the container with id 736fac086813d87ae3885bf3059fb75304a46b48e3d90ee1ea85c26b09e000bb
Nov 02 01:30:20.367564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.367472    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq" event={"ID":"5318f7c6-23a6-4de7-8f52-a54499d12bf9","Type":"ContainerStarted","Data":"736fac086813d87ae3885bf3059fb75304a46b48e3d90ee1ea85c26b09e000bb"}
Nov 02 01:30:20.670922 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.669412    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"]
Nov 02 01:30:20.688802 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.688125    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.743681 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.743649    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 01:30:20.743872 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.743754    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 01:30:20.747861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.747807    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 01:30:20.790435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.790381    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjf7g\""
Nov 02 01:30:20.791704 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.791633    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"]
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.798954    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.798997    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.799053    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.799082    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.799116    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.799181    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqbgt\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-kube-api-access-bqbgt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.799213    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.799242    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.799455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.799272    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.899860 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.899799    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqbgt\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-kube-api-access-bqbgt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.899860 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.899850    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.900114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.899879    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.900114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.899908    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.900114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.899941    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.900114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.899967    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.900114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.900021    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.900114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.900049    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.900114 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.900079    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.900556 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.900524    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.901120 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.901078    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.902535 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.902187    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.902535 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.902414    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.902535 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.902473    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.904904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.904869    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.906871 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.906847    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.976315 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.976215    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:20.977059 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:20.977032    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqbgt\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-kube-api-access-bqbgt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-98dt2\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:21.007403 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:21.007370    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:21.475413 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:21.475370    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"]
Nov 02 01:30:21.494159 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:21.494127    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1071dc1_e1dc_4ad1_ae82_91a80646a631.slice/crio-ac9f6d2d69d9dec4370bca2595c6b417f92b4c8c7b6ee372256ccf9795def9d4 WatchSource:0}: Error finding container ac9f6d2d69d9dec4370bca2595c6b417f92b4c8c7b6ee372256ccf9795def9d4: Status 404 returned error can't find the container with id ac9f6d2d69d9dec4370bca2595c6b417f92b4c8c7b6ee372256ccf9795def9d4
Nov 02 01:30:22.409747 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:22.409698    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" event={"ID":"d1071dc1-e1dc-4ad1-ae82-91a80646a631","Type":"ContainerStarted","Data":"ac9f6d2d69d9dec4370bca2595c6b417f92b4c8c7b6ee372256ccf9795def9d4"}
Nov 02 01:30:28.463540 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:28.463501    4287 generic.go:356] "Generic (PLEG): container finished" podID="2909310a-fff5-4192-afba-1f75800db215" containerID="3c683369f049c716c58e9f951f5a172c8a707525527bb26ff9aedcb7c197ebe6" exitCode=0
Nov 02 01:30:28.464051 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:28.463632    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl" event={"ID":"2909310a-fff5-4192-afba-1f75800db215","Type":"ContainerDied","Data":"3c683369f049c716c58e9f951f5a172c8a707525527bb26ff9aedcb7c197ebe6"}
Nov 02 01:30:29.355213 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:29.355175    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:29.355328 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:29.355263    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:29.355328 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:29.355307    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:30:29.476422 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:29.475492    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" event={"ID":"d1071dc1-e1dc-4ad1-ae82-91a80646a631","Type":"ContainerStarted","Data":"ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a"}
Nov 02 01:30:29.479090 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:29.479057    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq" event={"ID":"5318f7c6-23a6-4de7-8f52-a54499d12bf9","Type":"ContainerStarted","Data":"4890483cd6c3944a04eba9f3a60e4fa9b871760cf296740e358df817e0bf3c87"}
Nov 02 01:30:29.485215 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:29.485192    4287 generic.go:356] "Generic (PLEG): container finished" podID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerID="bd3ee71f68755c847212b16abe3f3aaa775652001d97d741b632dbbeb9cb51e6" exitCode=0
Nov 02 01:30:29.485322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:29.485253    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v" event={"ID":"2ae86c75-5bbf-49bd-a101-5962cd40b007","Type":"ContainerDied","Data":"bd3ee71f68755c847212b16abe3f3aaa775652001d97d741b632dbbeb9cb51e6"}
Nov 02 01:30:29.623770 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:29.622700    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" podStartSLOduration=1.766444115 podStartE2EDuration="9.622680271s" totalImagesPullingTime="7.856236156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:29.621830297 +0000 UTC m=+337.495471349" watchObservedRunningTime="2026-11-02 01:30:29.622680271 +0000 UTC m=+337.496321324"
Nov 02 01:30:30.008919 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:30.008835    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:30.014804 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:30.014545    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:30.492638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:30.492324    4287 generic.go:356] "Generic (PLEG): container finished" podID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerID="4890483cd6c3944a04eba9f3a60e4fa9b871760cf296740e358df817e0bf3c87" exitCode=0
Nov 02 01:30:30.495189 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:30.494851    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq" event={"ID":"5318f7c6-23a6-4de7-8f52-a54499d12bf9","Type":"ContainerDied","Data":"4890483cd6c3944a04eba9f3a60e4fa9b871760cf296740e358df817e0bf3c87"}
Nov 02 01:30:30.495415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:30.495303    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:30.496242 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:30.496217    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:30:32.503275 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.503230    4287 generic.go:356] "Generic (PLEG): container finished" podID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerID="956b85d96d5fe49ea592a88cb8b61bc65412135ea27861d1c6f5404ceb0acfa2" exitCode=0
Nov 02 01:30:32.503720 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.503359    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq" event={"ID":"5318f7c6-23a6-4de7-8f52-a54499d12bf9","Type":"ContainerDied","Data":"956b85d96d5fe49ea592a88cb8b61bc65412135ea27861d1c6f5404ceb0acfa2"}
Nov 02 01:30:32.506574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.506223    4287 generic.go:356] "Generic (PLEG): container finished" podID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerID="9bacca4bc5a59033a314ce483d21803b1b1ac7cd7918460979d5a99e1913ca26" exitCode=0
Nov 02 01:30:32.506574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.506350    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g" event={"ID":"a3fb3dc4-1a15-4290-905f-d413cd3d98f1","Type":"ContainerDied","Data":"9bacca4bc5a59033a314ce483d21803b1b1ac7cd7918460979d5a99e1913ca26"}
Nov 02 01:30:32.508272 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.508249    4287 generic.go:356] "Generic (PLEG): container finished" podID="be5e982b-e76a-454d-a38f-ac77343074a9" containerID="ec2ca6214b8f3d0dda88d4e4f1905e9ac330d454a393a585021dde9319db2567" exitCode=0
Nov 02 01:30:32.508368 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.508325    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb" event={"ID":"be5e982b-e76a-454d-a38f-ac77343074a9","Type":"ContainerDied","Data":"ec2ca6214b8f3d0dda88d4e4f1905e9ac330d454a393a585021dde9319db2567"}
Nov 02 01:30:32.510118 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.510085    4287 generic.go:356] "Generic (PLEG): container finished" podID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerID="5296014d9f7fa6bf199104252b6f42ab07dface3d70968550cdb34bd154873fb" exitCode=0
Nov 02 01:30:32.510213 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.510185    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm" event={"ID":"ec56b813-e200-4cd9-9d9a-0bded6d7644d","Type":"ContainerDied","Data":"5296014d9f7fa6bf199104252b6f42ab07dface3d70968550cdb34bd154873fb"}
Nov 02 01:30:32.511982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.511871    4287 generic.go:356] "Generic (PLEG): container finished" podID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerID="8d2847d9352ccf829aa952c95938507ea3789b1b5c2f863734432d41b2ce9c91" exitCode=0
Nov 02 01:30:32.511982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:32.511925    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp" event={"ID":"5cf780e8-26b5-4778-a680-8c08f2dca6a7","Type":"ContainerDied","Data":"8d2847d9352ccf829aa952c95938507ea3789b1b5c2f863734432d41b2ce9c91"}
Nov 02 01:30:33.518430 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:33.518393    4287 generic.go:356] "Generic (PLEG): container finished" podID="2909310a-fff5-4192-afba-1f75800db215" containerID="9bc8d6e74e9bfaedcebabfd2ad3de212892f137eede5618c36d3847c41d8fd1b" exitCode=0
Nov 02 01:30:33.518932 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:33.518501    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl" event={"ID":"2909310a-fff5-4192-afba-1f75800db215","Type":"ContainerDied","Data":"9bc8d6e74e9bfaedcebabfd2ad3de212892f137eede5618c36d3847c41d8fd1b"}
Nov 02 01:30:33.520517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:33.520491    4287 generic.go:356] "Generic (PLEG): container finished" podID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerID="b693b420b357a2eb00b3af622f5ed46ce6d56392c8db613018165239c1ef3777" exitCode=0
Nov 02 01:30:33.520652 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:33.520537    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v" event={"ID":"2ae86c75-5bbf-49bd-a101-5962cd40b007","Type":"ContainerDied","Data":"b693b420b357a2eb00b3af622f5ed46ce6d56392c8db613018165239c1ef3777"}
Nov 02 01:30:34.530158 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:34.530079    4287 generic.go:356] "Generic (PLEG): container finished" podID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerID="07441ac1f18cbf5e086e38ee83fd6398175a8b92d29ac7d71102188bc85690a2" exitCode=0
Nov 02 01:30:34.530580 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:34.530177    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm" event={"ID":"ec56b813-e200-4cd9-9d9a-0bded6d7644d","Type":"ContainerDied","Data":"07441ac1f18cbf5e086e38ee83fd6398175a8b92d29ac7d71102188bc85690a2"}
Nov 02 01:30:34.533056 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:34.532977    4287 generic.go:356] "Generic (PLEG): container finished" podID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerID="3fad6939d7e26f1f98bcbfc9ca185521aaa8700792b8371128972bedc63c14e9" exitCode=0
Nov 02 01:30:34.533056 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:34.533035    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp" event={"ID":"5cf780e8-26b5-4778-a680-8c08f2dca6a7","Type":"ContainerDied","Data":"3fad6939d7e26f1f98bcbfc9ca185521aaa8700792b8371128972bedc63c14e9"}
Nov 02 01:30:35.542362 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:35.542314    4287 generic.go:356] "Generic (PLEG): container finished" podID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerID="7fdbb43b06ccab734077a0f87b8e0b4a20a1132b4f7448df72e7859bf4de7058" exitCode=0
Nov 02 01:30:35.542864 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:35.542451    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g" event={"ID":"a3fb3dc4-1a15-4290-905f-d413cd3d98f1","Type":"ContainerDied","Data":"7fdbb43b06ccab734077a0f87b8e0b4a20a1132b4f7448df72e7859bf4de7058"}
Nov 02 01:30:43.602495 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:43.602453    4287 generic.go:356] "Generic (PLEG): container finished" podID="be5e982b-e76a-454d-a38f-ac77343074a9" containerID="239e7e7414b40d48bd36ee24e25b9046258405e54b31bfe360d3074af598cb47" exitCode=0
Nov 02 01:30:43.602985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:43.602530    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb" event={"ID":"be5e982b-e76a-454d-a38f-ac77343074a9","Type":"ContainerDied","Data":"239e7e7414b40d48bd36ee24e25b9046258405e54b31bfe360d3074af598cb47"}
Nov 02 01:30:45.620945 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:45.620851    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq" event={"ID":"5318f7c6-23a6-4de7-8f52-a54499d12bf9","Type":"ContainerStarted","Data":"c3e2fe024d143398ac90058cc0e18dd4a6fc588dff3f2f14d687e55ceb5e8e62"}
Nov 02 01:30:45.740371 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:45.740313    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq" podStartSLOduration=4.215562364 podStartE2EDuration="27.74029583s" totalImagesPullingTime="23.524733466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:45.739602095 +0000 UTC m=+353.613243149" watchObservedRunningTime="2026-11-02 01:30:45.74029583 +0000 UTC m=+353.613936882"
Nov 02 01:30:46.051751 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:30:46.051706    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:30:46.632208 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:46.632177    4287 generic.go:356] "Generic (PLEG): container finished" podID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerID="c3e2fe024d143398ac90058cc0e18dd4a6fc588dff3f2f14d687e55ceb5e8e62" exitCode=0
Nov 02 01:30:46.632695 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:46.632292    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq" event={"ID":"5318f7c6-23a6-4de7-8f52-a54499d12bf9","Type":"ContainerDied","Data":"c3e2fe024d143398ac90058cc0e18dd4a6fc588dff3f2f14d687e55ceb5e8e62"}
Nov 02 01:30:46.635648 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:46.635600    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v" event={"ID":"2ae86c75-5bbf-49bd-a101-5962cd40b007","Type":"ContainerStarted","Data":"6ab0556bacd6b80a788d2101d57ac3bc566ab7688757d97d4cef66eb3a071996"}
Nov 02 01:30:46.875210 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:46.874231    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v" podStartSLOduration=3.323136385 podStartE2EDuration="28.874211706s" totalImagesPullingTime="25.551075321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:46.873173157 +0000 UTC m=+354.746814257" watchObservedRunningTime="2026-11-02 01:30:46.874211706 +0000 UTC m=+354.747852759"
Nov 02 01:30:47.646335 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.646295    4287 generic.go:356] "Generic (PLEG): container finished" podID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerID="6ab0556bacd6b80a788d2101d57ac3bc566ab7688757d97d4cef66eb3a071996" exitCode=0
Nov 02 01:30:47.647054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.646985    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v" event={"ID":"2ae86c75-5bbf-49bd-a101-5962cd40b007","Type":"ContainerDied","Data":"6ab0556bacd6b80a788d2101d57ac3bc566ab7688757d97d4cef66eb3a071996"}
Nov 02 01:30:47.805519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.805305    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:47.858032 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.857686    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-bundle\") pod \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") "
Nov 02 01:30:47.858032 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.857751    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5318f7c6-23a6-4de7-8f52-a54499d12bf9-kube-api-access-zckj4\" (UniqueName: \"kubernetes.io/projected/5318f7c6-23a6-4de7-8f52-a54499d12bf9-kube-api-access-zckj4\") pod \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") "
Nov 02 01:30:47.858032 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.857807    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-util\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-util\") pod \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\" (UID: \"5318f7c6-23a6-4de7-8f52-a54499d12bf9\") "
Nov 02 01:30:47.859677 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.859639    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-bundle" pod "5318f7c6-23a6-4de7-8f52-a54499d12bf9" (UID: "5318f7c6-23a6-4de7-8f52-a54499d12bf9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:47.860747 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.860715    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5318f7c6-23a6-4de7-8f52-a54499d12bf9-kube-api-access-zckj4" pod "5318f7c6-23a6-4de7-8f52-a54499d12bf9" (UID: "5318f7c6-23a6-4de7-8f52-a54499d12bf9"). InnerVolumeSpecName "kube-api-access-zckj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:47.863132 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.863107    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-util" pod "5318f7c6-23a6-4de7-8f52-a54499d12bf9" (UID: "5318f7c6-23a6-4de7-8f52-a54499d12bf9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:47.959203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.959172    4287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:47.959203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.959201    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zckj4\" (UniqueName: \"kubernetes.io/projected/5318f7c6-23a6-4de7-8f52-a54499d12bf9-kube-api-access-zckj4\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:47.959203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:47.959212    4287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5318f7c6-23a6-4de7-8f52-a54499d12bf9-util\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:48.653903 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.653868    4287 generic.go:356] "Generic (PLEG): container finished" podID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerID="11044a2c4bb5825cb8d8111daecf7f7e04d881cbb9f19c526ea71067a3f3061b" exitCode=0
Nov 02 01:30:48.654434 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.653916    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm" event={"ID":"ec56b813-e200-4cd9-9d9a-0bded6d7644d","Type":"ContainerDied","Data":"11044a2c4bb5825cb8d8111daecf7f7e04d881cbb9f19c526ea71067a3f3061b"}
Nov 02 01:30:48.657263 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.657236    4287 generic.go:356] "Generic (PLEG): container finished" podID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerID="272ebcb4002b591698388bc37fa364b38101093becd12f076d9759111eb37d38" exitCode=0
Nov 02 01:30:48.657388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.657318    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp" event={"ID":"5cf780e8-26b5-4778-a680-8c08f2dca6a7","Type":"ContainerDied","Data":"272ebcb4002b591698388bc37fa364b38101093becd12f076d9759111eb37d38"}
Nov 02 01:30:48.660323 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.660302    4287 generic.go:356] "Generic (PLEG): container finished" podID="2909310a-fff5-4192-afba-1f75800db215" containerID="38868a14d06debaa14fb410bd81aaea999c9c97e9e972140406c51b0ef128da3" exitCode=0
Nov 02 01:30:48.660437 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.660381    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl" event={"ID":"2909310a-fff5-4192-afba-1f75800db215","Type":"ContainerDied","Data":"38868a14d06debaa14fb410bd81aaea999c9c97e9e972140406c51b0ef128da3"}
Nov 02 01:30:48.662900 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.662874    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq" event={"ID":"5318f7c6-23a6-4de7-8f52-a54499d12bf9","Type":"ContainerDied","Data":"736fac086813d87ae3885bf3059fb75304a46b48e3d90ee1ea85c26b09e000bb"}
Nov 02 01:30:48.663003 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.662913    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71zn6dq"
Nov 02 01:30:48.663003 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.662930    4287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="736fac086813d87ae3885bf3059fb75304a46b48e3d90ee1ea85c26b09e000bb"
Nov 02 01:30:48.825451 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.825428    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:48.970333 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.970249    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-util\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-util\") pod \"2ae86c75-5bbf-49bd-a101-5962cd40b007\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") "
Nov 02 01:30:48.970488 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.970349    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae86c75-5bbf-49bd-a101-5962cd40b007-kube-api-access-lc229\" (UniqueName: \"kubernetes.io/projected/2ae86c75-5bbf-49bd-a101-5962cd40b007-kube-api-access-lc229\") pod \"2ae86c75-5bbf-49bd-a101-5962cd40b007\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") "
Nov 02 01:30:48.970488 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.970423    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-bundle\") pod \"2ae86c75-5bbf-49bd-a101-5962cd40b007\" (UID: \"2ae86c75-5bbf-49bd-a101-5962cd40b007\") "
Nov 02 01:30:48.970914 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.970893    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-bundle" pod "2ae86c75-5bbf-49bd-a101-5962cd40b007" (UID: "2ae86c75-5bbf-49bd-a101-5962cd40b007"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:48.975098 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.975069    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-util" pod "2ae86c75-5bbf-49bd-a101-5962cd40b007" (UID: "2ae86c75-5bbf-49bd-a101-5962cd40b007"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:48.975978 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:48.975958    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae86c75-5bbf-49bd-a101-5962cd40b007-kube-api-access-lc229" pod "2ae86c75-5bbf-49bd-a101-5962cd40b007" (UID: "2ae86c75-5bbf-49bd-a101-5962cd40b007"). InnerVolumeSpecName "kube-api-access-lc229". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:49.071164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.071140    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc229\" (UniqueName: \"kubernetes.io/projected/2ae86c75-5bbf-49bd-a101-5962cd40b007-kube-api-access-lc229\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:49.071267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.071167    4287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:49.071267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.071176    4287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ae86c75-5bbf-49bd-a101-5962cd40b007-util\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:49.669351 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.669328    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v"
Nov 02 01:30:49.669816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.669320    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tdc4v" event={"ID":"2ae86c75-5bbf-49bd-a101-5962cd40b007","Type":"ContainerDied","Data":"2ac3fc1329754c78f4ba2a4fcf5cdafa46fdd30914b5b5cdb79a5cab8b592378"}
Nov 02 01:30:49.669816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.669457    4287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ac3fc1329754c78f4ba2a4fcf5cdafa46fdd30914b5b5cdb79a5cab8b592378"
Nov 02 01:30:49.671955 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.671930    4287 generic.go:356] "Generic (PLEG): container finished" podID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerID="0f079b69c071d47b9a1f12790d98cb746a3277924b42ef32718970cee1534992" exitCode=0
Nov 02 01:30:49.672087 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.672012    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g" event={"ID":"a3fb3dc4-1a15-4290-905f-d413cd3d98f1","Type":"ContainerDied","Data":"0f079b69c071d47b9a1f12790d98cb746a3277924b42ef32718970cee1534992"}
Nov 02 01:30:49.875827 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.875801    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:49.879445 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.879418    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:49.883195 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.883177    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:49.978779 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.978748    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-bundle\") pod \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") "
Nov 02 01:30:49.978920 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.978799    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-util\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-util\") pod \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") "
Nov 02 01:30:49.978920 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.978846    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec56b813-e200-4cd9-9d9a-0bded6d7644d-kube-api-access-td5v8\" (UniqueName: \"kubernetes.io/projected/ec56b813-e200-4cd9-9d9a-0bded6d7644d-kube-api-access-td5v8\") pod \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") "
Nov 02 01:30:49.978920 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.978899    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-util\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-util\") pod \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") "
Nov 02 01:30:49.979071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.978940    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-bundle\") pod \"2909310a-fff5-4192-afba-1f75800db215\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") "
Nov 02 01:30:49.979071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.978958    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf780e8-26b5-4778-a680-8c08f2dca6a7-kube-api-access-622s7\" (UniqueName: \"kubernetes.io/projected/5cf780e8-26b5-4778-a680-8c08f2dca6a7-kube-api-access-622s7\") pod \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\" (UID: \"5cf780e8-26b5-4778-a680-8c08f2dca6a7\") "
Nov 02 01:30:49.979071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.978983    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2909310a-fff5-4192-afba-1f75800db215-kube-api-access-pgjnz\" (UniqueName: \"kubernetes.io/projected/2909310a-fff5-4192-afba-1f75800db215-kube-api-access-pgjnz\") pod \"2909310a-fff5-4192-afba-1f75800db215\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") "
Nov 02 01:30:49.979071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.979011    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-bundle\") pod \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\" (UID: \"ec56b813-e200-4cd9-9d9a-0bded6d7644d\") "
Nov 02 01:30:49.979071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.979047    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-util\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-util\") pod \"2909310a-fff5-4192-afba-1f75800db215\" (UID: \"2909310a-fff5-4192-afba-1f75800db215\") "
Nov 02 01:30:49.979850 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.979823    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-bundle" pod "5cf780e8-26b5-4778-a680-8c08f2dca6a7" (UID: "5cf780e8-26b5-4778-a680-8c08f2dca6a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:49.980167 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.980105    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-bundle" pod "2909310a-fff5-4192-afba-1f75800db215" (UID: "2909310a-fff5-4192-afba-1f75800db215"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:49.981530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.981490    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec56b813-e200-4cd9-9d9a-0bded6d7644d-kube-api-access-td5v8" pod "ec56b813-e200-4cd9-9d9a-0bded6d7644d" (UID: "ec56b813-e200-4cd9-9d9a-0bded6d7644d"). InnerVolumeSpecName "kube-api-access-td5v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:49.981975 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.981941    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-bundle" pod "ec56b813-e200-4cd9-9d9a-0bded6d7644d" (UID: "ec56b813-e200-4cd9-9d9a-0bded6d7644d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:49.982160 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.982074    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2909310a-fff5-4192-afba-1f75800db215-kube-api-access-pgjnz" pod "2909310a-fff5-4192-afba-1f75800db215" (UID: "2909310a-fff5-4192-afba-1f75800db215"). InnerVolumeSpecName "kube-api-access-pgjnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:49.982220 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.982167    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf780e8-26b5-4778-a680-8c08f2dca6a7-kube-api-access-622s7" pod "5cf780e8-26b5-4778-a680-8c08f2dca6a7" (UID: "5cf780e8-26b5-4778-a680-8c08f2dca6a7"). InnerVolumeSpecName "kube-api-access-622s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:49.984541 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.984521    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-util" pod "2909310a-fff5-4192-afba-1f75800db215" (UID: "2909310a-fff5-4192-afba-1f75800db215"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:49.985360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.985339    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-util" pod "5cf780e8-26b5-4778-a680-8c08f2dca6a7" (UID: "5cf780e8-26b5-4778-a680-8c08f2dca6a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:49.986118 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:49.986097    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-util" pod "ec56b813-e200-4cd9-9d9a-0bded6d7644d" (UID: "ec56b813-e200-4cd9-9d9a-0bded6d7644d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:50.080171 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080149    4287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.080171 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080171    4287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cf780e8-26b5-4778-a680-8c08f2dca6a7-util\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.080298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080180    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td5v8\" (UniqueName: \"kubernetes.io/projected/ec56b813-e200-4cd9-9d9a-0bded6d7644d-kube-api-access-td5v8\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.080298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080189    4287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-util\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.080298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080198    4287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.080298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080225    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-622s7\" (UniqueName: \"kubernetes.io/projected/5cf780e8-26b5-4778-a680-8c08f2dca6a7-kube-api-access-622s7\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.080298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080234    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgjnz\" (UniqueName: \"kubernetes.io/projected/2909310a-fff5-4192-afba-1f75800db215-kube-api-access-pgjnz\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.080298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080242    4287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec56b813-e200-4cd9-9d9a-0bded6d7644d-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.080298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.080251    4287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2909310a-fff5-4192-afba-1f75800db215-util\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.677935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.677908    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm"
Nov 02 01:30:50.678367 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.677907    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sn2tm" event={"ID":"ec56b813-e200-4cd9-9d9a-0bded6d7644d","Type":"ContainerDied","Data":"90947635ffd3c1834ad59a34e05d989e1005a4118b90b1911e7802a8f3252b4e"}
Nov 02 01:30:50.678367 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.678038    4287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90947635ffd3c1834ad59a34e05d989e1005a4118b90b1911e7802a8f3252b4e"
Nov 02 01:30:50.680152 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.680131    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp" event={"ID":"5cf780e8-26b5-4778-a680-8c08f2dca6a7","Type":"ContainerDied","Data":"94f4d61878ea1d3a957a4a42c5cdfb254277a1ba7d1525498c9834f903f97df1"}
Nov 02 01:30:50.680152 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.680144    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acx7fp"
Nov 02 01:30:50.680307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.680163    4287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f4d61878ea1d3a957a4a42c5cdfb254277a1ba7d1525498c9834f903f97df1"
Nov 02 01:30:50.682308 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.682289    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl"
Nov 02 01:30:50.682308 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.682300    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck2xsl" event={"ID":"2909310a-fff5-4192-afba-1f75800db215","Type":"ContainerDied","Data":"2b8c9cce6ab69b475f2640af0b117d3e15089a5155f8779a396ca50418b067be"}
Nov 02 01:30:50.682440 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.682339    4287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b8c9cce6ab69b475f2640af0b117d3e15089a5155f8779a396ca50418b067be"
Nov 02 01:30:50.684571 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.684549    4287 generic.go:356] "Generic (PLEG): container finished" podID="be5e982b-e76a-454d-a38f-ac77343074a9" containerID="43ec2255dc81cc8d5186317467fb6f7f20669acfe44790261e6e419c3ad0d50e" exitCode=0
Nov 02 01:30:50.684676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.684633    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb" event={"ID":"be5e982b-e76a-454d-a38f-ac77343074a9","Type":"ContainerDied","Data":"43ec2255dc81cc8d5186317467fb6f7f20669acfe44790261e6e419c3ad0d50e"}
Nov 02 01:30:50.800346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.800324    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:50.884829 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.884764    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-kube-api-access-scf45\" (UniqueName: \"kubernetes.io/projected/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-kube-api-access-scf45\") pod \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") "
Nov 02 01:30:50.884829 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.884825    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-util\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-util\") pod \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") "
Nov 02 01:30:50.884995 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.884845    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-bundle\") pod \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\" (UID: \"a3fb3dc4-1a15-4290-905f-d413cd3d98f1\") "
Nov 02 01:30:50.885298 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.885272    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-bundle" pod "a3fb3dc4-1a15-4290-905f-d413cd3d98f1" (UID: "a3fb3dc4-1a15-4290-905f-d413cd3d98f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:50.886965 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.886943    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-kube-api-access-scf45" pod "a3fb3dc4-1a15-4290-905f-d413cd3d98f1" (UID: "a3fb3dc4-1a15-4290-905f-d413cd3d98f1"). InnerVolumeSpecName "kube-api-access-scf45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:50.890225 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.890205    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-util" pod "a3fb3dc4-1a15-4290-905f-d413cd3d98f1" (UID: "a3fb3dc4-1a15-4290-905f-d413cd3d98f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:50.986181 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.986159    4287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.986181 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.986180    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scf45\" (UniqueName: \"kubernetes.io/projected/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-kube-api-access-scf45\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:50.986317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:50.986189    4287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3fb3dc4-1a15-4290-905f-d413cd3d98f1-util\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:51.690119 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.690095    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g"
Nov 02 01:30:51.690512 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.690128    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nwr2g" event={"ID":"a3fb3dc4-1a15-4290-905f-d413cd3d98f1","Type":"ContainerDied","Data":"edb8833b3a0f1e749bd1dce3e7fb742cbb1a2bbe2508a46ba5f66d27f3d8d8de"}
Nov 02 01:30:51.690512 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.690171    4287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edb8833b3a0f1e749bd1dce3e7fb742cbb1a2bbe2508a46ba5f66d27f3d8d8de"
Nov 02 01:30:51.815919 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.815898    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:51.892228 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.892207    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5e982b-e76a-454d-a38f-ac77343074a9-kube-api-access-6x4kc\" (UniqueName: \"kubernetes.io/projected/be5e982b-e76a-454d-a38f-ac77343074a9-kube-api-access-6x4kc\") pod \"be5e982b-e76a-454d-a38f-ac77343074a9\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") "
Nov 02 01:30:51.892356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.892280    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-util\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-util\") pod \"be5e982b-e76a-454d-a38f-ac77343074a9\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") "
Nov 02 01:30:51.892356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.892302    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-bundle\") pod \"be5e982b-e76a-454d-a38f-ac77343074a9\" (UID: \"be5e982b-e76a-454d-a38f-ac77343074a9\") "
Nov 02 01:30:51.892863 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.892799    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-bundle" pod "be5e982b-e76a-454d-a38f-ac77343074a9" (UID: "be5e982b-e76a-454d-a38f-ac77343074a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:51.894267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.894233    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5e982b-e76a-454d-a38f-ac77343074a9-kube-api-access-6x4kc" pod "be5e982b-e76a-454d-a38f-ac77343074a9" (UID: "be5e982b-e76a-454d-a38f-ac77343074a9"). InnerVolumeSpecName "kube-api-access-6x4kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:30:51.897902 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.897883    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-util" pod "be5e982b-e76a-454d-a38f-ac77343074a9" (UID: "be5e982b-e76a-454d-a38f-ac77343074a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:30:51.993748 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.993694    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x4kc\" (UniqueName: \"kubernetes.io/projected/be5e982b-e76a-454d-a38f-ac77343074a9-kube-api-access-6x4kc\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:51.993748 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.993715    4287 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-util\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:51.993748 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:51.993724    4287 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be5e982b-e76a-454d-a38f-ac77343074a9-bundle\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:30:52.276334 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276302    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"]
Nov 02 01:30:52.276623 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276589    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2909310a-fff5-4192-afba-1f75800db215" containerName="util"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276626    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2909310a-fff5-4192-afba-1f75800db215" containerName="util"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276642    4287 cpu_manager.go:414] "containerMap: removing container" podUID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerName="pull"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276648    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerName="pull"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276660    4287 cpu_manager.go:414] "containerMap: removing container" podUID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerName="pull"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276666    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerName="pull"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276677    4287 cpu_manager.go:414] "containerMap: removing container" podUID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerName="pull"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276684    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerName="pull"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276696    4287 cpu_manager.go:414] "containerMap: removing container" podUID="be5e982b-e76a-454d-a38f-ac77343074a9" containerName="extract"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276703    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5e982b-e76a-454d-a38f-ac77343074a9" containerName="extract"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276712    4287 cpu_manager.go:414] "containerMap: removing container" podUID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerName="util"
Nov 02 01:30:52.276719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276719    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276733    4287 cpu_manager.go:414] "containerMap: removing container" podUID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276740    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276756    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerName="pull"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276763    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerName="pull"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276771    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2909310a-fff5-4192-afba-1f75800db215" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276778    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2909310a-fff5-4192-afba-1f75800db215" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276791    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276797    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276806    4287 cpu_manager.go:414] "containerMap: removing container" podUID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerName="pull"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276813    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerName="pull"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276823    4287 cpu_manager.go:414] "containerMap: removing container" podUID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276829    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276838    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2909310a-fff5-4192-afba-1f75800db215" containerName="pull"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276847    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2909310a-fff5-4192-afba-1f75800db215" containerName="pull"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276856    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276863    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276873    4287 cpu_manager.go:414] "containerMap: removing container" podUID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276879    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276892    4287 cpu_manager.go:414] "containerMap: removing container" podUID="be5e982b-e76a-454d-a38f-ac77343074a9" containerName="pull"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276899    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5e982b-e76a-454d-a38f-ac77343074a9" containerName="pull"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276910    4287 cpu_manager.go:414] "containerMap: removing container" podUID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276917    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276930    4287 cpu_manager.go:414] "containerMap: removing container" podUID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276937    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276947    4287 cpu_manager.go:414] "containerMap: removing container" podUID="be5e982b-e76a-454d-a38f-ac77343074a9" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276954    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5e982b-e76a-454d-a38f-ac77343074a9" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276970    4287 cpu_manager.go:414] "containerMap: removing container" podUID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276977    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerName="util"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276986    4287 cpu_manager.go:414] "containerMap: removing container" podUID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.276993    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.277059    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf780e8-26b5-4778-a680-8c08f2dca6a7" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.277072    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5e982b-e76a-454d-a38f-ac77343074a9" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.277082    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec56b813-e200-4cd9-9d9a-0bded6d7644d" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.277091    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2909310a-fff5-4192-afba-1f75800db215" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.277099    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae86c75-5bbf-49bd-a101-5962cd40b007" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.277109    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fb3dc4-1a15-4290-905f-d413cd3d98f1" containerName="extract"
Nov 02 01:30:52.277226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.277116    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5318f7c6-23a6-4de7-8f52-a54499d12bf9" containerName="extract"
Nov 02 01:30:52.279269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.279251    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.333515 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.333490    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 01:30:52.333719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.333703    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 01:30:52.333768 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.333758    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 01:30:52.334177 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.334160    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 01:30:52.334216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.334193    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-vxbpb\""
Nov 02 01:30:52.396425 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.396401    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.396522 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.396435    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.396522 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.396460    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.396629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.396570    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.396629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.396601    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfs4d\" (UniqueName: \"kubernetes.io/projected/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-kube-api-access-dfs4d\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.396704 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.396648    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.433484 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.433467    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 01:30:52.498020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.497966    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.498020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.498001    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.498166 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.498020    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.498166 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.498060    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.498166 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.498089    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfs4d\" (UniqueName: \"kubernetes.io/projected/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-kube-api-access-dfs4d\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.498166 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.498125    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.498459 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.498437    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.500303 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.500279    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.500566 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.500548    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.500642 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.500627    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.500700 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.500687    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.562952 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.562915    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"]
Nov 02 01:30:52.567214 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.567192    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfs4d\" (UniqueName: \"kubernetes.io/projected/a5c72e39-e792-45e8-be9e-bbb1fc9fde28-kube-api-access-dfs4d\") pod \"opendatahub-operator-controller-manager-5989ddd979-mc2br\" (UID: \"a5c72e39-e792-45e8-be9e-bbb1fc9fde28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.590656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.590639    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:52.695210 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.695185    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb"
Nov 02 01:30:52.695210 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.695186    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9cvqb" event={"ID":"be5e982b-e76a-454d-a38f-ac77343074a9","Type":"ContainerDied","Data":"4f8143594531d550793e0ac1d193919b5696e7829af2cd82951ccd43020765cf"}
Nov 02 01:30:52.695670 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:52.695242    4287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f8143594531d550793e0ac1d193919b5696e7829af2cd82951ccd43020765cf"
Nov 02 01:30:53.107483 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:53.107462    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"]
Nov 02 01:30:53.109646 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:30:53.109600    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c72e39_e792_45e8_be9e_bbb1fc9fde28.slice/crio-16e7909966fb33de1a0b45fef7a0cc41244e5b865962f2ddfaf30f5d43a71af4 WatchSource:0}: Error finding container 16e7909966fb33de1a0b45fef7a0cc41244e5b865962f2ddfaf30f5d43a71af4: Status 404 returned error can't find the container with id 16e7909966fb33de1a0b45fef7a0cc41244e5b865962f2ddfaf30f5d43a71af4
Nov 02 01:30:53.699907 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:53.699873    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br" event={"ID":"a5c72e39-e792-45e8-be9e-bbb1fc9fde28","Type":"ContainerStarted","Data":"16e7909966fb33de1a0b45fef7a0cc41244e5b865962f2ddfaf30f5d43a71af4"}
Nov 02 01:30:57.719427 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:57.719391    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br" event={"ID":"a5c72e39-e792-45e8-be9e-bbb1fc9fde28","Type":"ContainerStarted","Data":"26abaa13ad31885f9a7792d0dfbc1a1d73bb4936eaa6d4f74e777a900004a0cc"}
Nov 02 01:30:58.724878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:58.724842    4287 generic.go:356] "Generic (PLEG): container finished" podID="a5c72e39-e792-45e8-be9e-bbb1fc9fde28" containerID="26abaa13ad31885f9a7792d0dfbc1a1d73bb4936eaa6d4f74e777a900004a0cc" exitCode=0
Nov 02 01:30:58.724878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:58.724881    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br" event={"ID":"a5c72e39-e792-45e8-be9e-bbb1fc9fde28","Type":"ContainerDied","Data":"26abaa13ad31885f9a7792d0dfbc1a1d73bb4936eaa6d4f74e777a900004a0cc"}
Nov 02 01:30:59.507180 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.507153    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"]
Nov 02 01:30:59.509878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.509858    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"
Nov 02 01:30:59.562523 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.562500    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:30:59.562655 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.562550    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 01:30:59.562703 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.562690    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5m98l\""
Nov 02 01:30:59.606212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.606180    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"]
Nov 02 01:30:59.656817 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.656784    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbvv\" (UniqueName: \"kubernetes.io/projected/0bb19c54-2879-4284-893e-b5ac8e7e9f5a-kube-api-access-6mbvv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9\" (UID: \"0bb19c54-2879-4284-893e-b5ac8e7e9f5a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"
Nov 02 01:30:59.656952 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.656848    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb19c54-2879-4284-893e-b5ac8e7e9f5a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9\" (UID: \"0bb19c54-2879-4284-893e-b5ac8e7e9f5a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"
Nov 02 01:30:59.730486 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.730456    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br" event={"ID":"a5c72e39-e792-45e8-be9e-bbb1fc9fde28","Type":"ContainerStarted","Data":"7b23238edccea1597d23afc180bfb6b997eb6ac54b0644f822c1b9b347f854e1"}
Nov 02 01:30:59.730858 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.730589    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:30:59.758062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.758015    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb19c54-2879-4284-893e-b5ac8e7e9f5a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9\" (UID: \"0bb19c54-2879-4284-893e-b5ac8e7e9f5a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"
Nov 02 01:30:59.758120 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.758063    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbvv\" (UniqueName: \"kubernetes.io/projected/0bb19c54-2879-4284-893e-b5ac8e7e9f5a-kube-api-access-6mbvv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9\" (UID: \"0bb19c54-2879-4284-893e-b5ac8e7e9f5a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"
Nov 02 01:30:59.758402 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.758385    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb19c54-2879-4284-893e-b5ac8e7e9f5a-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9\" (UID: \"0bb19c54-2879-4284-893e-b5ac8e7e9f5a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"
Nov 02 01:30:59.827279 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.827247    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbvv\" (UniqueName: \"kubernetes.io/projected/0bb19c54-2879-4284-893e-b5ac8e7e9f5a-kube-api-access-6mbvv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9\" (UID: \"0bb19c54-2879-4284-893e-b5ac8e7e9f5a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"
Nov 02 01:30:59.843714 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.843694    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"
Nov 02 01:30:59.873404 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:30:59.873358    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br" podStartSLOduration=2.110983999 podStartE2EDuration="7.873343498s" totalImagesPullingTime="5.269973562s" totalInitContainerRuntime="492.385937ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:59.871166361 +0000 UTC m=+367.744807431" watchObservedRunningTime="2026-11-02 01:30:59.873343498 +0000 UTC m=+367.746984611"
Nov 02 01:31:00.265530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:00.265508    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9"]
Nov 02 01:31:00.268164 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:00.268135    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb19c54_2879_4284_893e_b5ac8e7e9f5a.slice/crio-7cf16770f2d952d7e80426e71aba9e3230af4b14295b71e377676b8f3398d5d9 WatchSource:0}: Error finding container 7cf16770f2d952d7e80426e71aba9e3230af4b14295b71e377676b8f3398d5d9: Status 404 returned error can't find the container with id 7cf16770f2d952d7e80426e71aba9e3230af4b14295b71e377676b8f3398d5d9
Nov 02 01:31:00.736043 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:00.735940    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9" event={"ID":"0bb19c54-2879-4284-893e-b5ac8e7e9f5a","Type":"ContainerStarted","Data":"7cf16770f2d952d7e80426e71aba9e3230af4b14295b71e377676b8f3398d5d9"}
Nov 02 01:31:03.437767 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.437688    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"]
Nov 02 01:31:03.439596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.439581    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"
Nov 02 01:31:03.493558 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.493535    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 01:31:03.493558 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.493551    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 01:31:03.499285 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.499253    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"]
Nov 02 01:31:03.590028 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.589996    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1178feb-c368-4e45-9f3c-e4749c92effb-tmp\") pod \"jobset-operator-674cd59dc5-j6x5j\" (UID: \"b1178feb-c368-4e45-9f3c-e4749c92effb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"
Nov 02 01:31:03.590164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.590123    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6ck\" (UniqueName: \"kubernetes.io/projected/b1178feb-c368-4e45-9f3c-e4749c92effb-kube-api-access-gh6ck\") pod \"jobset-operator-674cd59dc5-j6x5j\" (UID: \"b1178feb-c368-4e45-9f3c-e4749c92effb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"
Nov 02 01:31:03.593186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.593169    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-vbrjg\""
Nov 02 01:31:03.691210 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.691156    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6ck\" (UniqueName: \"kubernetes.io/projected/b1178feb-c368-4e45-9f3c-e4749c92effb-kube-api-access-gh6ck\") pod \"jobset-operator-674cd59dc5-j6x5j\" (UID: \"b1178feb-c368-4e45-9f3c-e4749c92effb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"
Nov 02 01:31:03.691210 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.691197    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1178feb-c368-4e45-9f3c-e4749c92effb-tmp\") pod \"jobset-operator-674cd59dc5-j6x5j\" (UID: \"b1178feb-c368-4e45-9f3c-e4749c92effb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"
Nov 02 01:31:03.691587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.691569    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1178feb-c368-4e45-9f3c-e4749c92effb-tmp\") pod \"jobset-operator-674cd59dc5-j6x5j\" (UID: \"b1178feb-c368-4e45-9f3c-e4749c92effb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"
Nov 02 01:31:03.750435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.750405    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9" event={"ID":"0bb19c54-2879-4284-893e-b5ac8e7e9f5a","Type":"ContainerStarted","Data":"9b3ace90f07a7f1c6b35c2a77625564a290ea6d35d0d5c00d6e0a4b503ca53c4"}
Nov 02 01:31:03.752265 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.752248    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6ck\" (UniqueName: \"kubernetes.io/projected/b1178feb-c368-4e45-9f3c-e4749c92effb-kube-api-access-gh6ck\") pod \"jobset-operator-674cd59dc5-j6x5j\" (UID: \"b1178feb-c368-4e45-9f3c-e4749c92effb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"
Nov 02 01:31:03.762034 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.762013    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"
Nov 02 01:31:03.876191 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:03.876143    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-m5jb9" podStartSLOduration=2.043452216 podStartE2EDuration="4.876128773s" totalImagesPullingTime="2.832676557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:03.875532752 +0000 UTC m=+371.749173804" watchObservedRunningTime="2026-11-02 01:31:03.876128773 +0000 UTC m=+371.749769825"
Nov 02 01:31:04.151702 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:04.151665    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j"]
Nov 02 01:31:04.153586 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:04.153562    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1178feb_c368_4e45_9f3c_e4749c92effb.slice/crio-b871252f53388f4afb5c1269c46ceb1a92dbe6789fb77713a1e3113f5928b8aa WatchSource:0}: Error finding container b871252f53388f4afb5c1269c46ceb1a92dbe6789fb77713a1e3113f5928b8aa: Status 404 returned error can't find the container with id b871252f53388f4afb5c1269c46ceb1a92dbe6789fb77713a1e3113f5928b8aa
Nov 02 01:31:04.756390 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:04.756353    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j" event={"ID":"b1178feb-c368-4e45-9f3c-e4749c92effb","Type":"ContainerStarted","Data":"b871252f53388f4afb5c1269c46ceb1a92dbe6789fb77713a1e3113f5928b8aa"}
Nov 02 01:31:07.773939 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:07.773902    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j" event={"ID":"b1178feb-c368-4e45-9f3c-e4749c92effb","Type":"ContainerStarted","Data":"b385d9b63c14778d9bc588b064a5c3e0c4538cb61a50a0ab4247308689a55910"}
Nov 02 01:31:07.890939 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:07.890888    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j6x5j" podStartSLOduration=1.96709945 podStartE2EDuration="4.890873833s" totalImagesPullingTime="2.923774383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:07.889479893 +0000 UTC m=+375.763120945" watchObservedRunningTime="2026-11-02 01:31:07.890873833 +0000 UTC m=+375.764514885"
Nov 02 01:31:08.448738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.448703    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"]
Nov 02 01:31:08.451855 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.451836    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"
Nov 02 01:31:08.504282 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.504249    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 01:31:08.504282 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.504259    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 01:31:08.504490 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.504429    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-f6cks\""
Nov 02 01:31:08.509878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.509848    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"]
Nov 02 01:31:08.529877 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.529850    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qb5\" (UniqueName: \"kubernetes.io/projected/c9d5d393-7b8f-4157-abb0-5601c81b1fe5-kube-api-access-x8qb5\") pod \"cert-manager-cainjector-5c57bbd4fc-p9lb8\" (UID: \"c9d5d393-7b8f-4157-abb0-5601c81b1fe5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"
Nov 02 01:31:08.529975 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.529894    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d5d393-7b8f-4157-abb0-5601c81b1fe5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p9lb8\" (UID: \"c9d5d393-7b8f-4157-abb0-5601c81b1fe5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"
Nov 02 01:31:08.630581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.630554    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qb5\" (UniqueName: \"kubernetes.io/projected/c9d5d393-7b8f-4157-abb0-5601c81b1fe5-kube-api-access-x8qb5\") pod \"cert-manager-cainjector-5c57bbd4fc-p9lb8\" (UID: \"c9d5d393-7b8f-4157-abb0-5601c81b1fe5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"
Nov 02 01:31:08.630751 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.630596    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d5d393-7b8f-4157-abb0-5601c81b1fe5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p9lb8\" (UID: \"c9d5d393-7b8f-4157-abb0-5601c81b1fe5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"
Nov 02 01:31:08.692376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.692345    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d5d393-7b8f-4157-abb0-5601c81b1fe5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-p9lb8\" (UID: \"c9d5d393-7b8f-4157-abb0-5601c81b1fe5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"
Nov 02 01:31:08.692526 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.692470    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qb5\" (UniqueName: \"kubernetes.io/projected/c9d5d393-7b8f-4157-abb0-5601c81b1fe5-kube-api-access-x8qb5\") pod \"cert-manager-cainjector-5c57bbd4fc-p9lb8\" (UID: \"c9d5d393-7b8f-4157-abb0-5601c81b1fe5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"
Nov 02 01:31:08.762014 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:08.761991    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"
Nov 02 01:31:09.133102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:09.133024    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8"]
Nov 02 01:31:09.135243 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:09.135218    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9d5d393_7b8f_4157_abb0_5601c81b1fe5.slice/crio-d3aac2dc6b6b9a110d8b633503ffe31afd62ae2629ba2aa5d9a5bcb62d0ed8ed WatchSource:0}: Error finding container d3aac2dc6b6b9a110d8b633503ffe31afd62ae2629ba2aa5d9a5bcb62d0ed8ed: Status 404 returned error can't find the container with id d3aac2dc6b6b9a110d8b633503ffe31afd62ae2629ba2aa5d9a5bcb62d0ed8ed
Nov 02 01:31:09.784139 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:09.784105    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8" event={"ID":"c9d5d393-7b8f-4157-abb0-5601c81b1fe5","Type":"ContainerStarted","Data":"d3aac2dc6b6b9a110d8b633503ffe31afd62ae2629ba2aa5d9a5bcb62d0ed8ed"}
Nov 02 01:31:10.738652 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:10.738601    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-mc2br"
Nov 02 01:31:13.802654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:13.802623    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8" event={"ID":"c9d5d393-7b8f-4157-abb0-5601c81b1fe5","Type":"ContainerStarted","Data":"cb03fd499ca7b3308becf211839a212af89021ea602e1aae919e8aa2b7e1dd2b"}
Nov 02 01:31:13.826757 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:13.826726    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"]
Nov 02 01:31:13.830112 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:13.830090    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:13.883386 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:13.883356    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-b84xj\""
Nov 02 01:31:13.909385 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:13.909359    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"]
Nov 02 01:31:13.938550 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:13.938505    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-p9lb8" podStartSLOduration=2.001944211 podStartE2EDuration="5.938492785s" totalImagesPullingTime="3.936548574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:13.937083165 +0000 UTC m=+381.810724218" watchObservedRunningTime="2026-11-02 01:31:13.938492785 +0000 UTC m=+381.812133836"
Nov 02 01:31:13.975317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:13.975281    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4c9d\" (UniqueName: \"kubernetes.io/projected/e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6-kube-api-access-z4c9d\") pod \"cert-manager-webhook-5dbbbf75cc-2d4kg\" (UID: \"e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:13.975475 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:13.975377    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2d4kg\" (UID: \"e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:14.076244 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:14.076160    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2d4kg\" (UID: \"e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:14.076244 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:14.076225    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4c9d\" (UniqueName: \"kubernetes.io/projected/e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6-kube-api-access-z4c9d\") pod \"cert-manager-webhook-5dbbbf75cc-2d4kg\" (UID: \"e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:14.142201 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:14.142167    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2d4kg\" (UID: \"e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:14.142356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:14.142326    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4c9d\" (UniqueName: \"kubernetes.io/projected/e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6-kube-api-access-z4c9d\") pod \"cert-manager-webhook-5dbbbf75cc-2d4kg\" (UID: \"e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:14.442117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:14.442037    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:14.816305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:14.816283    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"]
Nov 02 01:31:14.818433 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:14.818408    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode72048b4_e5e8_4c4f_a3f5_c9a1847f7ee6.slice/crio-f8739d9b3512eb2277296f0e4e994e666368270d08eaa4bb9b4f89cf7a638622 WatchSource:0}: Error finding container f8739d9b3512eb2277296f0e4e994e666368270d08eaa4bb9b4f89cf7a638622: Status 404 returned error can't find the container with id f8739d9b3512eb2277296f0e4e994e666368270d08eaa4bb9b4f89cf7a638622
Nov 02 01:31:15.812100 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:15.812058    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg" event={"ID":"e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6","Type":"ContainerStarted","Data":"2a327ed2059942ae2dad3624740151afef31a94240dfde135466d0db3442e32d"}
Nov 02 01:31:15.812276 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:15.812109    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg" event={"ID":"e72048b4-e5e8-4c4f-a3f5-c9a1847f7ee6","Type":"ContainerStarted","Data":"f8739d9b3512eb2277296f0e4e994e666368270d08eaa4bb9b4f89cf7a638622"}
Nov 02 01:31:15.812276 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:15.812136    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:15.927884 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:15.927833    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg" podStartSLOduration=2.927818775 podStartE2EDuration="2.927818775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:15.927459355 +0000 UTC m=+383.801100424" watchObservedRunningTime="2026-11-02 01:31:15.927818775 +0000 UTC m=+383.801459827"
Nov 02 01:31:16.037202 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:31:16.037170    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:31:21.818679 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:21.818645    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2d4kg"
Nov 02 01:31:24.998738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:24.998699    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"]
Nov 02 01:31:25.003235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.003216    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"
Nov 02 01:31:25.105811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.105776    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 01:31:25.105811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.105781    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 01:31:25.105811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.105781    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 01:31:25.106033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.105824    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-js4r5\""
Nov 02 01:31:25.112383 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.112355    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"]
Nov 02 01:31:25.155274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.155235    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfd9c\" (UniqueName: \"kubernetes.io/projected/30ca19e2-0eef-41b4-b291-aa3d46a5f9ac-kube-api-access-pfd9c\") pod \"dns-operator-controller-manager-8557cbdb84-wxbbm\" (UID: \"30ca19e2-0eef-41b4-b291-aa3d46a5f9ac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"
Nov 02 01:31:25.256678 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.256568    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfd9c\" (UniqueName: \"kubernetes.io/projected/30ca19e2-0eef-41b4-b291-aa3d46a5f9ac-kube-api-access-pfd9c\") pod \"dns-operator-controller-manager-8557cbdb84-wxbbm\" (UID: \"30ca19e2-0eef-41b4-b291-aa3d46a5f9ac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"
Nov 02 01:31:25.326396 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.326365    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfd9c\" (UniqueName: \"kubernetes.io/projected/30ca19e2-0eef-41b4-b291-aa3d46a5f9ac-kube-api-access-pfd9c\") pod \"dns-operator-controller-manager-8557cbdb84-wxbbm\" (UID: \"30ca19e2-0eef-41b4-b291-aa3d46a5f9ac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"
Nov 02 01:31:25.615980 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.615951    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"
Nov 02 01:31:25.986024 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:25.985990    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"]
Nov 02 01:31:25.988540 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:25.988514    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ca19e2_0eef_41b4_b291_aa3d46a5f9ac.slice/crio-da09f0390820058169ff1c88aacb680f46d4a9205c1b33375b53a9a4240980a4 WatchSource:0}: Error finding container da09f0390820058169ff1c88aacb680f46d4a9205c1b33375b53a9a4240980a4: Status 404 returned error can't find the container with id da09f0390820058169ff1c88aacb680f46d4a9205c1b33375b53a9a4240980a4
Nov 02 01:31:26.859806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:26.859777    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm" event={"ID":"30ca19e2-0eef-41b4-b291-aa3d46a5f9ac","Type":"ContainerStarted","Data":"da09f0390820058169ff1c88aacb680f46d4a9205c1b33375b53a9a4240980a4"}
Nov 02 01:31:30.879980 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:30.879891    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm" event={"ID":"30ca19e2-0eef-41b4-b291-aa3d46a5f9ac","Type":"ContainerStarted","Data":"c1850a036d28e749348499f2acfd6048f32228708c8218c5832130cd1db08046"}
Nov 02 01:31:30.879980 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:30.879967    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"
Nov 02 01:31:31.000249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:31.000191    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm" podStartSLOduration=2.47545386 podStartE2EDuration="7.000175643s" totalImagesPullingTime="4.524721783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:30.998894754 +0000 UTC m=+398.872535807" watchObservedRunningTime="2026-11-02 01:31:31.000175643 +0000 UTC m=+398.873816695"
Nov 02 01:31:38.840545 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.840458    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79f447648c-49sv8"]
Nov 02 01:31:38.842761 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.842739    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:38.856338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.856312    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9edb95bd-6855-44f2-b04c-716f90f13af7-tls-certs\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:38.856464 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.856372    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvmv\" (UniqueName: \"kubernetes.io/projected/9edb95bd-6855-44f2-b04c-716f90f13af7-kube-api-access-wvvmv\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:38.856518 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.856464    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9edb95bd-6855-44f2-b04c-716f90f13af7-tmp\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:38.894926 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.894900    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 01:31:38.944222 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.944197    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qv7s4\""
Nov 02 01:31:38.944366 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.944276    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 01:31:38.947390 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.947363    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79f447648c-49sv8"]
Nov 02 01:31:38.957292 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.957271    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvmv\" (UniqueName: \"kubernetes.io/projected/9edb95bd-6855-44f2-b04c-716f90f13af7-kube-api-access-wvvmv\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:38.957394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.957325    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9edb95bd-6855-44f2-b04c-716f90f13af7-tmp\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:38.957394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.957377    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9edb95bd-6855-44f2-b04c-716f90f13af7-tls-certs\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:38.959957 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.959935    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9edb95bd-6855-44f2-b04c-716f90f13af7-tmp\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:38.960264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:38.960245    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9edb95bd-6855-44f2-b04c-716f90f13af7-tls-certs\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:39.018230 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.018204    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvmv\" (UniqueName: \"kubernetes.io/projected/9edb95bd-6855-44f2-b04c-716f90f13af7-kube-api-access-wvvmv\") pod \"kube-auth-proxy-79f447648c-49sv8\" (UID: \"9edb95bd-6855-44f2-b04c-716f90f13af7\") " pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:39.152753 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.152676    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8"
Nov 02 01:31:39.389368 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.389328    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"]
Nov 02 01:31:39.391502 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.391480    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.455040 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.454970    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"]
Nov 02 01:31:39.462123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462093    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.462244 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462203    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.462244 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462231    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a24b8b-043b-4029-bff2-a91708e86173-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.462345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462254    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.462345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462329    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgd6\" (UniqueName: \"kubernetes.io/projected/d5a24b8b-043b-4029-bff2-a91708e86173-kube-api-access-zsgd6\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.462476 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462458    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a24b8b-043b-4029-bff2-a91708e86173-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.462518 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462500    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.462560 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462550    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.462701 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.462685    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a24b8b-043b-4029-bff2-a91708e86173-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.487576 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.487556    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nlmb9\""
Nov 02 01:31:39.520868 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.520850    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79f447648c-49sv8"]
Nov 02 01:31:39.523305 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:39.523281    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edb95bd_6855_44f2_b04c_716f90f13af7.slice/crio-a3e37d2d055064e5602b8f5f71351f855672923ee4feeb09e438335230812508 WatchSource:0}: Error finding container a3e37d2d055064e5602b8f5f71351f855672923ee4feeb09e438335230812508: Status 404 returned error can't find the container with id a3e37d2d055064e5602b8f5f71351f855672923ee4feeb09e438335230812508
Nov 02 01:31:39.563157 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563130    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563167    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a24b8b-043b-4029-bff2-a91708e86173-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563200    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563252    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563276    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a24b8b-043b-4029-bff2-a91708e86173-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563306    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563339    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgd6\" (UniqueName: \"kubernetes.io/projected/d5a24b8b-043b-4029-bff2-a91708e86173-kube-api-access-zsgd6\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563397    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a24b8b-043b-4029-bff2-a91708e86173-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563642 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563446    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563638    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563685    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.563962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563930    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.564122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.563961    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.564122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.564107    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a24b8b-043b-4029-bff2-a91708e86173-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.565702 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.565675    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a24b8b-043b-4029-bff2-a91708e86173-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.565795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.565780    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a24b8b-043b-4029-bff2-a91708e86173-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.623930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.623906    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a24b8b-043b-4029-bff2-a91708e86173-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.624103 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.624086    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgd6\" (UniqueName: \"kubernetes.io/projected/d5a24b8b-043b-4029-bff2-a91708e86173-kube-api-access-zsgd6\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-xnjzm\" (UID: \"d5a24b8b-043b-4029-bff2-a91708e86173\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.703636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.703593    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:39.922805 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:39.922771    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8" event={"ID":"9edb95bd-6855-44f2-b04c-716f90f13af7","Type":"ContainerStarted","Data":"a3e37d2d055064e5602b8f5f71351f855672923ee4feeb09e438335230812508"}
Nov 02 01:31:40.127172 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:40.127146    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"]
Nov 02 01:31:40.128656 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:40.128629    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a24b8b_043b_4029_bff2_a91708e86173.slice/crio-e6ac6917fef8e053bc83b646e72adad7421b58a15478ecc2276c7927f8b7f9d6 WatchSource:0}: Error finding container e6ac6917fef8e053bc83b646e72adad7421b58a15478ecc2276c7927f8b7f9d6: Status 404 returned error can't find the container with id e6ac6917fef8e053bc83b646e72adad7421b58a15478ecc2276c7927f8b7f9d6
Nov 02 01:31:40.130800 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:40.130766    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:40.130872 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:40.130847    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:40.130909 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:40.130887    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:31:40.931900 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:40.931858    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm" event={"ID":"d5a24b8b-043b-4029-bff2-a91708e86173","Type":"ContainerStarted","Data":"64f2db2274f7dc68f7e98643c23426a4fc273fedce84273efe22fca4516e7ab8"}
Nov 02 01:31:40.932442 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:40.931917    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm" event={"ID":"d5a24b8b-043b-4029-bff2-a91708e86173","Type":"ContainerStarted","Data":"e6ac6917fef8e053bc83b646e72adad7421b58a15478ecc2276c7927f8b7f9d6"}
Nov 02 01:31:41.102628 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:41.102554    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm" podStartSLOduration=2.102538809 podStartE2EDuration="2.102538809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:41.101681775 +0000 UTC m=+408.975322827" watchObservedRunningTime="2026-11-02 01:31:41.102538809 +0000 UTC m=+408.976179922"
Nov 02 01:31:41.704366 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:41.704331    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:41.710382 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:41.710357    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:41.890831 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:41.890795    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-wxbbm"
Nov 02 01:31:41.936682 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:41.936649    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:41.937521 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:41.937502    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-xnjzm"
Nov 02 01:31:42.171859 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.171826    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"]
Nov 02 01:31:42.182746 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.182716    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:42.236549 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.236375    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 01:31:42.245118 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.245091    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"]
Nov 02 01:31:42.285906 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.285877    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdt2\" (UniqueName: \"kubernetes.io/projected/8e59af5c-98e4-4104-bd51-fb85f881b912-kube-api-access-btdt2\") pod \"limitador-limitador-74dfcfb894-cb5wk\" (UID: \"8e59af5c-98e4-4104-bd51-fb85f881b912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:42.286098 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.285918    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e59af5c-98e4-4104-bd51-fb85f881b912-config-file\") pod \"limitador-limitador-74dfcfb894-cb5wk\" (UID: \"8e59af5c-98e4-4104-bd51-fb85f881b912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:42.311661 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.311629    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8jvk2\""
Nov 02 01:31:42.386675 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.386649    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e59af5c-98e4-4104-bd51-fb85f881b912-config-file\") pod \"limitador-limitador-74dfcfb894-cb5wk\" (UID: \"8e59af5c-98e4-4104-bd51-fb85f881b912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:42.386825 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.386733    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdt2\" (UniqueName: \"kubernetes.io/projected/8e59af5c-98e4-4104-bd51-fb85f881b912-kube-api-access-btdt2\") pod \"limitador-limitador-74dfcfb894-cb5wk\" (UID: \"8e59af5c-98e4-4104-bd51-fb85f881b912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:42.387269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.387250    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e59af5c-98e4-4104-bd51-fb85f881b912-config-file\") pod \"limitador-limitador-74dfcfb894-cb5wk\" (UID: \"8e59af5c-98e4-4104-bd51-fb85f881b912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:42.416247 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.416214    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"]
Nov 02 01:31:42.450941 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.450875    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdt2\" (UniqueName: \"kubernetes.io/projected/8e59af5c-98e4-4104-bd51-fb85f881b912-kube-api-access-btdt2\") pod \"limitador-limitador-74dfcfb894-cb5wk\" (UID: \"8e59af5c-98e4-4104-bd51-fb85f881b912\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:42.497412 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.497389    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:42.881713 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.881690    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"]
Nov 02 01:31:42.883345 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:42.883320    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e59af5c_98e4_4104_bd51_fb85f881b912.slice/crio-771b7070955ed7b5247a614a02b455773e5dd6aecba14db712f7342b1b8a073e WatchSource:0}: Error finding container 771b7070955ed7b5247a614a02b455773e5dd6aecba14db712f7342b1b8a073e: Status 404 returned error can't find the container with id 771b7070955ed7b5247a614a02b455773e5dd6aecba14db712f7342b1b8a073e
Nov 02 01:31:42.941351 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.941325    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk" event={"ID":"8e59af5c-98e4-4104-bd51-fb85f881b912","Type":"ContainerStarted","Data":"771b7070955ed7b5247a614a02b455773e5dd6aecba14db712f7342b1b8a073e"}
Nov 02 01:31:42.942954 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:42.942844    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8" event={"ID":"9edb95bd-6855-44f2-b04c-716f90f13af7","Type":"ContainerStarted","Data":"2c7900c59d1cfda5fe093144229246f110e8a7347a4e31042454efab9c024162"}
Nov 02 01:31:43.066638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:43.066520    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79f447648c-49sv8" podStartSLOduration=1.793076661 podStartE2EDuration="5.066500307s" totalImagesPullingTime="3.273423646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:43.064051546 +0000 UTC m=+410.937692598" watchObservedRunningTime="2026-11-02 01:31:43.066500307 +0000 UTC m=+410.940141363"
Nov 02 01:31:45.958225 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:45.958155    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk" event={"ID":"8e59af5c-98e4-4104-bd51-fb85f881b912","Type":"ContainerStarted","Data":"09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28"}
Nov 02 01:31:45.958560 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:45.958280    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:31:46.038144 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:31:46.038107    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:31:46.079469 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.079421    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk" podStartSLOduration=1.440817466 podStartE2EDuration="4.079406646s" totalImagesPullingTime="2.63858918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:46.079284973 +0000 UTC m=+413.952926025" watchObservedRunningTime="2026-11-02 01:31:46.079406646 +0000 UTC m=+413.953047697"
Nov 02 01:31:46.656737 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.656700    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"]
Nov 02 01:31:46.662393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.660277    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.717313 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.717279    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"]
Nov 02 01:31:46.718136 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.718107    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 01:31:46.723569 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.723546    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f10be300-f0c2-485d-bea2-c8cd85512179-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.723691 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.723597    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf9kk\" (UniqueName: \"kubernetes.io/projected/f10be300-f0c2-485d-bea2-c8cd85512179-kube-api-access-wf9kk\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.723733 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.723701    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f10be300-f0c2-485d-bea2-c8cd85512179-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.764679 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.764647    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 01:31:46.825074 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.825045    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f10be300-f0c2-485d-bea2-c8cd85512179-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.825230 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.825112    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf9kk\" (UniqueName: \"kubernetes.io/projected/f10be300-f0c2-485d-bea2-c8cd85512179-kube-api-access-wf9kk\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.825230 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.825142    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f10be300-f0c2-485d-bea2-c8cd85512179-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.825773 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.825751    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f10be300-f0c2-485d-bea2-c8cd85512179-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.827392 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.827370    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f10be300-f0c2-485d-bea2-c8cd85512179-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.885554 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.885525    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf9kk\" (UniqueName: \"kubernetes.io/projected/f10be300-f0c2-485d-bea2-c8cd85512179-kube-api-access-wf9kk\") pod \"kuadrant-console-plugin-66c4fb575d-44hpr\" (UID: \"f10be300-f0c2-485d-bea2-c8cd85512179\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:46.974404 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:46.974335    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"
Nov 02 01:31:47.341703 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:47.341577    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr"]
Nov 02 01:31:47.344174 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:31:47.344149    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf10be300_f0c2_485d_bea2_c8cd85512179.slice/crio-10eaae6f9946b903b74444c630c27a2a95b476163f8ee5966d87c3bf0d95aa12 WatchSource:0}: Error finding container 10eaae6f9946b903b74444c630c27a2a95b476163f8ee5966d87c3bf0d95aa12: Status 404 returned error can't find the container with id 10eaae6f9946b903b74444c630c27a2a95b476163f8ee5966d87c3bf0d95aa12
Nov 02 01:31:47.968807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:47.968773    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr" event={"ID":"f10be300-f0c2-485d-bea2-c8cd85512179","Type":"ContainerStarted","Data":"10eaae6f9946b903b74444c630c27a2a95b476163f8ee5966d87c3bf0d95aa12"}
Nov 02 01:31:56.006669 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:56.006630    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr" event={"ID":"f10be300-f0c2-485d-bea2-c8cd85512179","Type":"ContainerStarted","Data":"1a4359acb0d4367b4015eeeee4eea4e9a00398c3938bb82b422877f8a6dd957e"}
Nov 02 01:31:56.121021 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:56.120935    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-44hpr" podStartSLOduration=1.673591067 podStartE2EDuration="10.12092211s" totalImagesPullingTime="8.447331043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:56.120417793 +0000 UTC m=+423.994058846" watchObservedRunningTime="2026-11-02 01:31:56.12092211 +0000 UTC m=+423.994563162"
Nov 02 01:31:56.964576 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:31:56.964546    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:32:00.876458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:00.876421    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"]
Nov 02 01:32:00.876994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:00.876705    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk" podUID="8e59af5c-98e4-4104-bd51-fb85f881b912" containerName="limitador" containerID="cri-o://09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28" gracePeriod=30
Nov 02 01:32:01.414407 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:01.414385    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:32:01.548573 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:01.548543    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e59af5c-98e4-4104-bd51-fb85f881b912-config-file\" (UniqueName: \"kubernetes.io/configmap/8e59af5c-98e4-4104-bd51-fb85f881b912-config-file\") pod \"8e59af5c-98e4-4104-bd51-fb85f881b912\" (UID: \"8e59af5c-98e4-4104-bd51-fb85f881b912\") "
Nov 02 01:32:01.548764 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:01.548585    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e59af5c-98e4-4104-bd51-fb85f881b912-kube-api-access-btdt2\" (UniqueName: \"kubernetes.io/projected/8e59af5c-98e4-4104-bd51-fb85f881b912-kube-api-access-btdt2\") pod \"8e59af5c-98e4-4104-bd51-fb85f881b912\" (UID: \"8e59af5c-98e4-4104-bd51-fb85f881b912\") "
Nov 02 01:32:01.549028 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:01.548998    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e59af5c-98e4-4104-bd51-fb85f881b912-config-file" pod "8e59af5c-98e4-4104-bd51-fb85f881b912" (UID: "8e59af5c-98e4-4104-bd51-fb85f881b912"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:32:01.550977 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:01.550952    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e59af5c-98e4-4104-bd51-fb85f881b912-kube-api-access-btdt2" pod "8e59af5c-98e4-4104-bd51-fb85f881b912" (UID: "8e59af5c-98e4-4104-bd51-fb85f881b912"). InnerVolumeSpecName "kube-api-access-btdt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:01.649517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:01.649472    4287 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8e59af5c-98e4-4104-bd51-fb85f881b912-config-file\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:01.649517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:01.649504    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdt2\" (UniqueName: \"kubernetes.io/projected/8e59af5c-98e4-4104-bd51-fb85f881b912-kube-api-access-btdt2\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:02.035288 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.035254    4287 generic.go:356] "Generic (PLEG): container finished" podID="8e59af5c-98e4-4104-bd51-fb85f881b912" containerID="09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28" exitCode=0
Nov 02 01:32:02.035815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.035299    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"
Nov 02 01:32:02.035815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.035348    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk" event={"ID":"8e59af5c-98e4-4104-bd51-fb85f881b912","Type":"ContainerDied","Data":"09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28"}
Nov 02 01:32:02.035815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.035411    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-cb5wk" event={"ID":"8e59af5c-98e4-4104-bd51-fb85f881b912","Type":"ContainerDied","Data":"771b7070955ed7b5247a614a02b455773e5dd6aecba14db712f7342b1b8a073e"}
Nov 02 01:32:02.035815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.035448    4287 scope.go:122] "RemoveContainer" containerID="09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28"
Nov 02 01:32:02.044333 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.044310    4287 scope.go:122] "RemoveContainer" containerID="09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28"
Nov 02 01:32:02.044574 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:32:02.044558    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28\": container with ID starting with 09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28 not found: ID does not exist" containerID="09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28"
Nov 02 01:32:02.044648 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.044582    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28"} err="failed to get container status \"09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28\": rpc error: code = NotFound desc = could not find container \"09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28\": container with ID starting with 09123d07c765f889ae67ef7469f72dd904b04a1c0cb10d120f5193afc02f9c28 not found: ID does not exist"
Nov 02 01:32:02.198539 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.198504    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"]
Nov 02 01:32:02.248008 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:02.247972    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-cb5wk"]
Nov 02 01:32:04.236453 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.236421    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e59af5c-98e4-4104-bd51-fb85f881b912" path="/var/lib/kubelet/pods/8e59af5c-98e4-4104-bd51-fb85f881b912/volumes"
Nov 02 01:32:04.875694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.875664    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"]
Nov 02 01:32:04.875994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.875982    4287 cpu_manager.go:414] "containerMap: removing container" podUID="8e59af5c-98e4-4104-bd51-fb85f881b912" containerName="limitador"
Nov 02 01:32:04.876046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.875995    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e59af5c-98e4-4104-bd51-fb85f881b912" containerName="limitador"
Nov 02 01:32:04.876080 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.876059    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e59af5c-98e4-4104-bd51-fb85f881b912" containerName="limitador"
Nov 02 01:32:04.880085 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.880069    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:04.936221 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.936188    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"]
Nov 02 01:32:04.980921 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.980886    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2896\" (UniqueName: \"kubernetes.io/projected/cc5970e6-3332-4454-b542-290d640ae6fc-kube-api-access-p2896\") pod \"postgres-7cdbf6dc44-2fwct\" (UID: \"cc5970e6-3332-4454-b542-290d640ae6fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:04.981077 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.980931    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc5970e6-3332-4454-b542-290d640ae6fc-data\") pod \"postgres-7cdbf6dc44-2fwct\" (UID: \"cc5970e6-3332-4454-b542-290d640ae6fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:04.983534 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.983516    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 01:32:04.983534 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.983517    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-z2fmm\""
Nov 02 01:32:04.983534 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.983517    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 01:32:04.983744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:04.983574    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 01:32:05.082155 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.082106    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2896\" (UniqueName: \"kubernetes.io/projected/cc5970e6-3332-4454-b542-290d640ae6fc-kube-api-access-p2896\") pod \"postgres-7cdbf6dc44-2fwct\" (UID: \"cc5970e6-3332-4454-b542-290d640ae6fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:05.082358 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.082170    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc5970e6-3332-4454-b542-290d640ae6fc-data\") pod \"postgres-7cdbf6dc44-2fwct\" (UID: \"cc5970e6-3332-4454-b542-290d640ae6fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:05.082595 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.082571    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cc5970e6-3332-4454-b542-290d640ae6fc-data\") pod \"postgres-7cdbf6dc44-2fwct\" (UID: \"cc5970e6-3332-4454-b542-290d640ae6fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:05.143088 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.143007    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2896\" (UniqueName: \"kubernetes.io/projected/cc5970e6-3332-4454-b542-290d640ae6fc-kube-api-access-p2896\") pod \"postgres-7cdbf6dc44-2fwct\" (UID: \"cc5970e6-3332-4454-b542-290d640ae6fc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:05.191239 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.191208    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:05.338937 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.338905    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"]
Nov 02 01:32:05.341274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.341246    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.393547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.393457    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-xrhc4\""
Nov 02 01:32:05.393547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.393516    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 01:32:05.393547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.393519    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 01:32:05.486123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.486094    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0da1661-c27c-4a4b-9331-87ff64a33683-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.486123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.486128    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0da1661-c27c-4a4b-9331-87ff64a33683-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.486328 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.486220    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0da1661-c27c-4a4b-9331-87ff64a33683-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.486373 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.486350    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshwh\" (UniqueName: \"kubernetes.io/projected/b0da1661-c27c-4a4b-9331-87ff64a33683-kube-api-access-lshwh\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.493117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.493083    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 01:32:05.495208 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.495187    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"]
Nov 02 01:32:05.560669 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.560649    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"]
Nov 02 01:32:05.562804 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:32:05.562773    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5970e6_3332_4454_b542_290d640ae6fc.slice/crio-7a057a6a728975396ceec23e0abb90683444fcfb3839d7eb80499a1fa7157716 WatchSource:0}: Error finding container 7a057a6a728975396ceec23e0abb90683444fcfb3839d7eb80499a1fa7157716: Status 404 returned error can't find the container with id 7a057a6a728975396ceec23e0abb90683444fcfb3839d7eb80499a1fa7157716
Nov 02 01:32:05.587230 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.587204    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0da1661-c27c-4a4b-9331-87ff64a33683-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.587354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.587248    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshwh\" (UniqueName: \"kubernetes.io/projected/b0da1661-c27c-4a4b-9331-87ff64a33683-kube-api-access-lshwh\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.587444 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.587422    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0da1661-c27c-4a4b-9331-87ff64a33683-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.587521 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.587457    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0da1661-c27c-4a4b-9331-87ff64a33683-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.587667 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.587649    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b0da1661-c27c-4a4b-9331-87ff64a33683-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.587748 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.587731    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0da1661-c27c-4a4b-9331-87ff64a33683-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.588188 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.588171    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0da1661-c27c-4a4b-9331-87ff64a33683-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.648010 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.647943    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshwh\" (UniqueName: \"kubernetes.io/projected/b0da1661-c27c-4a4b-9331-87ff64a33683-kube-api-access-lshwh\") pod \"kserve-module-controller-manager-7566cf5fc8-tc4gc\" (UID: \"b0da1661-c27c-4a4b-9331-87ff64a33683\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:05.654722 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:05.654697    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:06.055966 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:06.055931    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct" event={"ID":"cc5970e6-3332-4454-b542-290d640ae6fc","Type":"ContainerStarted","Data":"7a057a6a728975396ceec23e0abb90683444fcfb3839d7eb80499a1fa7157716"}
Nov 02 01:32:06.075225 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:06.075192    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"]
Nov 02 01:32:06.077453 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:32:06.077428    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0da1661_c27c_4a4b_9331_87ff64a33683.slice/crio-047ac3379c3466b8108cd724cf72fdfa146be911d9258fe683165c3094d9feed WatchSource:0}: Error finding container 047ac3379c3466b8108cd724cf72fdfa146be911d9258fe683165c3094d9feed: Status 404 returned error can't find the container with id 047ac3379c3466b8108cd724cf72fdfa146be911d9258fe683165c3094d9feed
Nov 02 01:32:07.061974 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:07.061939    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc" event={"ID":"b0da1661-c27c-4a4b-9331-87ff64a33683","Type":"ContainerStarted","Data":"047ac3379c3466b8108cd724cf72fdfa146be911d9258fe683165c3094d9feed"}
Nov 02 01:32:09.076249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:09.076210    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc" event={"ID":"b0da1661-c27c-4a4b-9331-87ff64a33683","Type":"ContainerStarted","Data":"af9bd9f7d5514592ca597c08532d5decbb0e9710610f6d9e9067362081125e05"}
Nov 02 01:32:09.076751 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:09.076366    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:09.244354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:09.244298    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc" podStartSLOduration=1.853487629 podStartE2EDuration="4.244282601s" totalImagesPullingTime="2.390794972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:09.243915323 +0000 UTC m=+437.117556376" watchObservedRunningTime="2026-11-02 01:32:09.244282601 +0000 UTC m=+437.117923654"
Nov 02 01:32:12.093178 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:12.093141    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct" event={"ID":"cc5970e6-3332-4454-b542-290d640ae6fc","Type":"ContainerStarted","Data":"d9f550c8a9fdbf5e6f7d39114dcdeb135131fb4f77cbaae92a132b984178a09f"}
Nov 02 01:32:12.093566 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:12.093304    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:12.207743 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:12.207651    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct" podStartSLOduration=1.884990827 podStartE2EDuration="8.20763846s" totalImagesPullingTime="6.322647633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:12.207401484 +0000 UTC m=+440.081042536" watchObservedRunningTime="2026-11-02 01:32:12.20763846 +0000 UTC m=+440.081279509"
Nov 02 01:32:16.038028 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:32:16.037996    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:32:18.127479 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:18.127453    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fwct"
Nov 02 01:32:25.307318 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.307282    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bkc5p"]
Nov 02 01:32:25.315544 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.315521    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.393212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.393190    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 01:32:25.393212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.393191    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 01:32:25.393388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.393225    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 01:32:25.417020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.416991    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghnpp\""
Nov 02 01:32:25.418985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.418963    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bkc5p"]
Nov 02 01:32:25.442653 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.442631    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-webhook-cert\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.442820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.442750    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-metrics-certs\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.442820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.442795    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxc4n\" (UniqueName: \"kubernetes.io/projected/8690239a-ceae-4c8f-a8a0-6937d83f5aff-kube-api-access-vxc4n\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.543484 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.543447    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-metrics-certs\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.543663 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.543490    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxc4n\" (UniqueName: \"kubernetes.io/projected/8690239a-ceae-4c8f-a8a0-6937d83f5aff-kube-api-access-vxc4n\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.543739 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.543669    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-webhook-cert\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.546095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.546070    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-metrics-certs\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.546195 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.546113    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-webhook-cert\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.604232 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.604167    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxc4n\" (UniqueName: \"kubernetes.io/projected/8690239a-ceae-4c8f-a8a0-6937d83f5aff-kube-api-access-vxc4n\") pod \"maas-controller-579cf7c6d-bkc5p\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") " pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:25.630616 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:25.630586    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:26.009440 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:26.009416    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bkc5p"]
Nov 02 01:32:26.011932 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:32:26.011909    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8690239a_ceae_4c8f_a8a0_6937d83f5aff.slice/crio-07db3bdd57d23b59c03eb804cc26c4b55b16b86488bead43fefa4df15d4c8951 WatchSource:0}: Error finding container 07db3bdd57d23b59c03eb804cc26c4b55b16b86488bead43fefa4df15d4c8951: Status 404 returned error can't find the container with id 07db3bdd57d23b59c03eb804cc26c4b55b16b86488bead43fefa4df15d4c8951
Nov 02 01:32:26.157130 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:26.157101    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-bkc5p" event={"ID":"8690239a-ceae-4c8f-a8a0-6937d83f5aff","Type":"ContainerStarted","Data":"07db3bdd57d23b59c03eb804cc26c4b55b16b86488bead43fefa4df15d4c8951"}
Nov 02 01:32:29.174112 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:29.174075    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-bkc5p" event={"ID":"8690239a-ceae-4c8f-a8a0-6937d83f5aff","Type":"ContainerStarted","Data":"a9f629f69f0756afd7026972f4e5d5b6c3e61619311a30e261e2a1ebee0b62bf"}
Nov 02 01:32:29.174513 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:29.174218    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:29.307049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:29.306988    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-bkc5p" podStartSLOduration=1.237764827 podStartE2EDuration="4.306974229s" totalImagesPullingTime="3.069209402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:29.305307846 +0000 UTC m=+457.178948900" watchObservedRunningTime="2026-11-02 01:32:29.306974229 +0000 UTC m=+457.180615281"
Nov 02 01:32:30.082275 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:30.082243    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tc4gc"
Nov 02 01:32:32.440838 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.440808    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"]
Nov 02 01:32:32.443368 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.443350    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.495117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.495093    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 02 01:32:32.495117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.495108    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 02 01:32:32.500803 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.500781    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.501001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.500975    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.501121 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.501016    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.501185 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.501157    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdqhn\" (UniqueName: \"kubernetes.io/projected/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-kube-api-access-cdqhn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.544763 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.544724    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 02 01:32:32.544872 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.544763    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-c2nht\""
Nov 02 01:32:32.548041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.548025    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"]
Nov 02 01:32:32.601512 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.601479    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdqhn\" (UniqueName: \"kubernetes.io/projected/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-kube-api-access-cdqhn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.601651 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.601523    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.601651 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.601559    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.601651 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.601577    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.601771 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:32:32.601665    4287 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 02 01:32:32.601771 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:32:32.601694    4287 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 02 01:32:32.601771 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:32:32.601731    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-proxy-source podName:e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e nodeName:}" failed. No retries permitted until 2026-11-02 01:32:33.101715082 +0000 UTC m=+460.975356128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-ww2c9" (UID: "e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e") : configmap "usage-tenancy-proxy-source" not found
Nov 02 01:32:32.601771 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:32:32.601768    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-tls-cert podName:e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e nodeName:}" failed. No retries permitted until 2026-11-02 01:32:33.101742003 +0000 UTC m=+460.975383049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-ww2c9" (UID: "e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 02 01:32:32.602255 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.602238    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:32.662716 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:32.662688    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdqhn\" (UniqueName: \"kubernetes.io/projected/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-kube-api-access-cdqhn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:33.104376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:33.104347    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:33.104538 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:33.104397    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:33.105198 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:33.105173    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:33.106741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:33.106722    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ww2c9\" (UID: \"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:33.354449 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:33.354368    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:33.725450 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:33.725425    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"]
Nov 02 01:32:33.727013 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:32:33.726987    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode591d4a5_c7fd_4ed6_bd4f_e0498a1ffd2e.slice/crio-4867f221ec8df9af9efd831de82fdc97c06d5418eef1040bc725083c446a0f4a WatchSource:0}: Error finding container 4867f221ec8df9af9efd831de82fdc97c06d5418eef1040bc725083c446a0f4a: Status 404 returned error can't find the container with id 4867f221ec8df9af9efd831de82fdc97c06d5418eef1040bc725083c446a0f4a
Nov 02 01:32:34.197417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:34.197382    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9" event={"ID":"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e","Type":"ContainerStarted","Data":"4867f221ec8df9af9efd831de82fdc97c06d5418eef1040bc725083c446a0f4a"}
Nov 02 01:32:40.181431 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:40.181397    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:41.059575 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.059537    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bkc5p"]
Nov 02 01:32:41.059969 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.059812    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-bkc5p" podUID="8690239a-ceae-4c8f-a8a0-6937d83f5aff" containerName="manager" containerID="cri-o://a9f629f69f0756afd7026972f4e5d5b6c3e61619311a30e261e2a1ebee0b62bf" gracePeriod=30
Nov 02 01:32:41.260768 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.260654    4287 generic.go:356] "Generic (PLEG): container finished" podID="8690239a-ceae-4c8f-a8a0-6937d83f5aff" containerID="a9f629f69f0756afd7026972f4e5d5b6c3e61619311a30e261e2a1ebee0b62bf" exitCode=0
Nov 02 01:32:41.261318 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.261245    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-bkc5p" event={"ID":"8690239a-ceae-4c8f-a8a0-6937d83f5aff","Type":"ContainerDied","Data":"a9f629f69f0756afd7026972f4e5d5b6c3e61619311a30e261e2a1ebee0b62bf"}
Nov 02 01:32:41.348410 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.348379    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:41.467688 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.465280    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8690239a-ceae-4c8f-a8a0-6937d83f5aff-kube-api-access-vxc4n\" (UniqueName: \"kubernetes.io/projected/8690239a-ceae-4c8f-a8a0-6937d83f5aff-kube-api-access-vxc4n\") pod \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") "
Nov 02 01:32:41.467688 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.465373    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-metrics-certs\") pod \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") "
Nov 02 01:32:41.467688 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.465459    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-webhook-cert\") pod \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\" (UID: \"8690239a-ceae-4c8f-a8a0-6937d83f5aff\") "
Nov 02 01:32:41.472261 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.472197    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8690239a-ceae-4c8f-a8a0-6937d83f5aff-kube-api-access-vxc4n" pod "8690239a-ceae-4c8f-a8a0-6937d83f5aff" (UID: "8690239a-ceae-4c8f-a8a0-6937d83f5aff"). InnerVolumeSpecName "kube-api-access-vxc4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:32:41.472550 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.472487    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-metrics-certs" pod "8690239a-ceae-4c8f-a8a0-6937d83f5aff" (UID: "8690239a-ceae-4c8f-a8a0-6937d83f5aff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:41.485553 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.485524    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-webhook-cert" pod "8690239a-ceae-4c8f-a8a0-6937d83f5aff" (UID: "8690239a-ceae-4c8f-a8a0-6937d83f5aff"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:32:41.566097 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.566062    4287 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-webhook-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:41.566097 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.566092    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxc4n\" (UniqueName: \"kubernetes.io/projected/8690239a-ceae-4c8f-a8a0-6937d83f5aff-kube-api-access-vxc4n\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:41.566097 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:41.566102    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8690239a-ceae-4c8f-a8a0-6937d83f5aff-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:32:42.268380 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:42.268351    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-bkc5p" event={"ID":"8690239a-ceae-4c8f-a8a0-6937d83f5aff","Type":"ContainerDied","Data":"07db3bdd57d23b59c03eb804cc26c4b55b16b86488bead43fefa4df15d4c8951"}
Nov 02 01:32:42.268848 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:42.268401    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-bkc5p"
Nov 02 01:32:42.268848 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:42.268408    4287 scope.go:122] "RemoveContainer" containerID="a9f629f69f0756afd7026972f4e5d5b6c3e61619311a30e261e2a1ebee0b62bf"
Nov 02 01:32:42.432026 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:42.431955    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bkc5p"]
Nov 02 01:32:42.481306 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:42.481269    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-bkc5p"]
Nov 02 01:32:44.238218 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:44.238187    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8690239a-ceae-4c8f-a8a0-6937d83f5aff" path="/var/lib/kubelet/pods/8690239a-ceae-4c8f-a8a0-6937d83f5aff/volumes"
Nov 02 01:32:46.039293 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:32:46.039251    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:32:49.312926 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:49.312885    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9" event={"ID":"e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e","Type":"ContainerStarted","Data":"916f3cf1e1d7785a28b6b81b3bd54d90f99246569822da8c0ba76ba8726c481f"}
Nov 02 01:32:49.313425 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:49.312949    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:32:49.430716 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:32:49.430664    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9" podStartSLOduration=2.598301569 podStartE2EDuration="17.430650017s" totalImagesPullingTime="14.832348448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:32:49.428723201 +0000 UTC m=+477.302364253" watchObservedRunningTime="2026-11-02 01:32:49.430650017 +0000 UTC m=+477.304291065"
Nov 02 01:33:00.323232 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:00.323201    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ww2c9"
Nov 02 01:33:02.460691 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.460651    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"]
Nov 02 01:33:02.461198 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.461178    4287 cpu_manager.go:414] "containerMap: removing container" podUID="8690239a-ceae-4c8f-a8a0-6937d83f5aff" containerName="manager"
Nov 02 01:33:02.461241 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.461202    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8690239a-ceae-4c8f-a8a0-6937d83f5aff" containerName="manager"
Nov 02 01:33:02.461314 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.461300    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="8690239a-ceae-4c8f-a8a0-6937d83f5aff" containerName="manager"
Nov 02 01:33:02.467546 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.467521    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.528760 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.528722    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-metrics-certs\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.528936 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.528774    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5cad2c-4851-4f0f-a04e-c70f832210c6-openshift-service-ca\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.528936 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.528857    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-maas-api-tls\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.528936 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.528909    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pk4s\" (UniqueName: \"kubernetes.io/projected/9f5cad2c-4851-4f0f-a04e-c70f832210c6-kube-api-access-4pk4s\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.575059 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.575026    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nk57n\""
Nov 02 01:33:02.575236 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.575026    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 01:33:02.575236 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.575146    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 01:33:02.578354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.578327    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"]
Nov 02 01:33:02.629516 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.629477    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-metrics-certs\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.629516 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.629522    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5cad2c-4851-4f0f-a04e-c70f832210c6-openshift-service-ca\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.629762 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.629544    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-maas-api-tls\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.629762 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.629586    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pk4s\" (UniqueName: \"kubernetes.io/projected/9f5cad2c-4851-4f0f-a04e-c70f832210c6-kube-api-access-4pk4s\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.630339 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.630262    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5cad2c-4851-4f0f-a04e-c70f832210c6-openshift-service-ca\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.632188 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.632166    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-metrics-certs\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.632281 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.632231    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-maas-api-tls\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.691233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.691203    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pk4s\" (UniqueName: \"kubernetes.io/projected/9f5cad2c-4851-4f0f-a04e-c70f832210c6-kube-api-access-4pk4s\") pod \"maas-api-6d669c7fcd-hq9m7\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:02.780223 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:02.780193    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:03.197506 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:03.197481    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"]
Nov 02 01:33:03.199694 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:33:03.199669    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5cad2c_4851_4f0f_a04e_c70f832210c6.slice/crio-2ff9744b5a3344f696c60d89524e71082766d0ff878c51b60f5a26ad69cc77e8 WatchSource:0}: Error finding container 2ff9744b5a3344f696c60d89524e71082766d0ff878c51b60f5a26ad69cc77e8: Status 404 returned error can't find the container with id 2ff9744b5a3344f696c60d89524e71082766d0ff878c51b60f5a26ad69cc77e8
Nov 02 01:33:03.380803 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:03.380764    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7" event={"ID":"9f5cad2c-4851-4f0f-a04e-c70f832210c6","Type":"ContainerStarted","Data":"2ff9744b5a3344f696c60d89524e71082766d0ff878c51b60f5a26ad69cc77e8"}
Nov 02 01:33:05.391928 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:05.391894    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7" event={"ID":"9f5cad2c-4851-4f0f-a04e-c70f832210c6","Type":"ContainerStarted","Data":"c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de"}
Nov 02 01:33:05.392346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:05.391962    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:05.577711 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:05.577574    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7" podStartSLOduration=1.462483935 podStartE2EDuration="3.577559698s" totalImagesPullingTime="2.115075763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:05.577042039 +0000 UTC m=+493.450683090" watchObservedRunningTime="2026-11-02 01:33:05.577559698 +0000 UTC m=+493.451200747"
Nov 02 01:33:11.402415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:11.402382    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:12.441880 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.441846    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"]
Nov 02 01:33:12.475126 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.475094    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.505253 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.505226    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"]
Nov 02 01:33:12.515894 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.515869    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eead6b71-0ae4-4224-a254-42d2b6008d0c-openshift-service-ca\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.516047 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.515909    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtfk\" (UniqueName: \"kubernetes.io/projected/eead6b71-0ae4-4224-a254-42d2b6008d0c-kube-api-access-7wtfk\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.516047 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.515936    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-metrics-certs\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.516178 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.516153    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-maas-api-tls\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.617503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.617469    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eead6b71-0ae4-4224-a254-42d2b6008d0c-openshift-service-ca\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.617503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.617506    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtfk\" (UniqueName: \"kubernetes.io/projected/eead6b71-0ae4-4224-a254-42d2b6008d0c-kube-api-access-7wtfk\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.617776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.617524    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-metrics-certs\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.617776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.617671    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-maas-api-tls\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.618322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.618280    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eead6b71-0ae4-4224-a254-42d2b6008d0c-openshift-service-ca\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.620341 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.620306    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-metrics-certs\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.620529 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.620503    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-maas-api-tls\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.679291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.679263    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtfk\" (UniqueName: \"kubernetes.io/projected/eead6b71-0ae4-4224-a254-42d2b6008d0c-kube-api-access-7wtfk\") pod \"maas-api-f5655ff5-xfn62\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") " pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:12.830392 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:12.830356    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:13.215093 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.215007    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"]
Nov 02 01:33:13.218098 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.218075    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:13.274671 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.250472    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"]
Nov 02 01:33:13.274671 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.271947    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 01:33:13.274671 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.272156    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6dchm\""
Nov 02 01:33:13.278075 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:33:13.278049    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeead6b71_0ae4_4224_a254_42d2b6008d0c.slice/crio-97bb288be44acebea86d7cb858563e521c075fa3ecff177d18c4e7cfe1395794 WatchSource:0}: Error finding container 97bb288be44acebea86d7cb858563e521c075fa3ecff177d18c4e7cfe1395794: Status 404 returned error can't find the container with id 97bb288be44acebea86d7cb858563e521c075fa3ecff177d18c4e7cfe1395794
Nov 02 01:33:13.281301 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.281278    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"]
Nov 02 01:33:13.324350 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.324315    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80d48db7-565a-4d7a-92b5-1825b3ba74ce-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tr9r9\" (UID: \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:13.324487 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.324376    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqsl\" (UniqueName: \"kubernetes.io/projected/80d48db7-565a-4d7a-92b5-1825b3ba74ce-kube-api-access-6rqsl\") pod \"llmisvc-controller-manager-869d5dcc5d-tr9r9\" (UID: \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:13.425502 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.425471    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqsl\" (UniqueName: \"kubernetes.io/projected/80d48db7-565a-4d7a-92b5-1825b3ba74ce-kube-api-access-6rqsl\") pod \"llmisvc-controller-manager-869d5dcc5d-tr9r9\" (UID: \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:13.425735 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.425526    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80d48db7-565a-4d7a-92b5-1825b3ba74ce-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tr9r9\" (UID: \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:13.428242 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.428214    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80d48db7-565a-4d7a-92b5-1825b3ba74ce-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tr9r9\" (UID: \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:13.428638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.428618    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62" event={"ID":"eead6b71-0ae4-4224-a254-42d2b6008d0c","Type":"ContainerStarted","Data":"97bb288be44acebea86d7cb858563e521c075fa3ecff177d18c4e7cfe1395794"}
Nov 02 01:33:13.486580 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.486505    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqsl\" (UniqueName: \"kubernetes.io/projected/80d48db7-565a-4d7a-92b5-1825b3ba74ce-kube-api-access-6rqsl\") pod \"llmisvc-controller-manager-869d5dcc5d-tr9r9\" (UID: \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:13.530601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.530574    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:13.927798 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:13.927769    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"]
Nov 02 01:33:13.929884 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:33:13.929859    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80d48db7_565a_4d7a_92b5_1825b3ba74ce.slice/crio-e3e01ccbb7f0553aee2ebab4313ea6b54c9c6de9ee69831579d9293e0f698c07 WatchSource:0}: Error finding container e3e01ccbb7f0553aee2ebab4313ea6b54c9c6de9ee69831579d9293e0f698c07: Status 404 returned error can't find the container with id e3e01ccbb7f0553aee2ebab4313ea6b54c9c6de9ee69831579d9293e0f698c07
Nov 02 01:33:14.436244 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:14.436139    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62" event={"ID":"eead6b71-0ae4-4224-a254-42d2b6008d0c","Type":"ContainerStarted","Data":"bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f"}
Nov 02 01:33:14.436424 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:14.436298    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:14.437661 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:14.437629    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9" event={"ID":"80d48db7-565a-4d7a-92b5-1825b3ba74ce","Type":"ContainerStarted","Data":"e3e01ccbb7f0553aee2ebab4313ea6b54c9c6de9ee69831579d9293e0f698c07"}
Nov 02 01:33:14.555080 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:14.555016    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62" podStartSLOduration=1.7438598490000001 podStartE2EDuration="2.554998163s" totalImagesPullingTime="811.138314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:14.55433959 +0000 UTC m=+502.427980644" watchObservedRunningTime="2026-11-02 01:33:14.554998163 +0000 UTC m=+502.428639215"
Nov 02 01:33:16.044800 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:33:16.044758    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:33:16.538132 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:16.537807    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-98dt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:33:16.538132 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:16.537877    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:17.454525 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.454421    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9" event={"ID":"80d48db7-565a-4d7a-92b5-1825b3ba74ce","Type":"ContainerStarted","Data":"8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c"}
Nov 02 01:33:17.454525 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.454492    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:33:17.456063 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.456035    4287 generic.go:356] "Generic (PLEG): container finished" podID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerID="ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a" exitCode=0
Nov 02 01:33:17.456175 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.456084    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" event={"ID":"d1071dc1-e1dc-4ad1-ae82-91a80646a631","Type":"ContainerDied","Data":"ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a"}
Nov 02 01:33:17.456372 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.456361    4287 scope.go:122] "RemoveContainer" containerID="ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a"
Nov 02 01:33:17.457352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.457325    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:17.457442 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.457390    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:17.457442 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.457422    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:33:17.572193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:17.572123    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9" podStartSLOduration=1.814367121 podStartE2EDuration="4.572105149s" totalImagesPullingTime="2.757738028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:17.570307231 +0000 UTC m=+505.443948283" watchObservedRunningTime="2026-11-02 01:33:17.572105149 +0000 UTC m=+505.445746206"
Nov 02 01:33:18.477375 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:18.477339    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" event={"ID":"d1071dc1-e1dc-4ad1-ae82-91a80646a631","Type":"ContainerStarted","Data":"ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637"}
Nov 02 01:33:19.008028 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:19.008001    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:33:19.008274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:19.008178    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-98dt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Nov 02 01:33:19.008274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:19.008229    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Nov 02 01:33:20.008982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:20.008943    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-98dt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Nov 02 01:33:20.009530 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:20.009004    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Nov 02 01:33:20.449020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:20.448676    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 01:33:20.626398 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:20.626364    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"]
Nov 02 01:33:20.626661 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:20.626637    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7" podUID="9f5cad2c-4851-4f0f-a04e-c70f832210c6" containerName="maas-api" containerID="cri-o://c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de" gracePeriod=30
Nov 02 01:33:21.008742 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.008662    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-98dt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Nov 02 01:33:21.008742 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.008723    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Nov 02 01:33:21.442732 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.442704    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:21.493411 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.493374    4287 generic.go:356] "Generic (PLEG): container finished" podID="9f5cad2c-4851-4f0f-a04e-c70f832210c6" containerID="c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de" exitCode=0
Nov 02 01:33:21.493574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.493432    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"
Nov 02 01:33:21.493574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.493455    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7" event={"ID":"9f5cad2c-4851-4f0f-a04e-c70f832210c6","Type":"ContainerDied","Data":"c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de"}
Nov 02 01:33:21.493574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.493518    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7" event={"ID":"9f5cad2c-4851-4f0f-a04e-c70f832210c6","Type":"ContainerDied","Data":"2ff9744b5a3344f696c60d89524e71082766d0ff878c51b60f5a26ad69cc77e8"}
Nov 02 01:33:21.493574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.493547    4287 scope.go:122] "RemoveContainer" containerID="c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de"
Nov 02 01:33:21.502328 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.502302    4287 scope.go:122] "RemoveContainer" containerID="c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de"
Nov 02 01:33:21.502592 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:33:21.502573    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de\": container with ID starting with c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de not found: ID does not exist" containerID="c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de"
Nov 02 01:33:21.502753 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.502600    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de"} err="failed to get container status \"c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de\": rpc error: code = NotFound desc = could not find container \"c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de\": container with ID starting with c4b22ed18b8f333dbdbd3dd6d2d2e17c9443910dc06180ecc73c80a29cd1b6de not found: ID does not exist"
Nov 02 01:33:21.505888 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.505872    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5cad2c-4851-4f0f-a04e-c70f832210c6-kube-api-access-4pk4s\" (UniqueName: \"kubernetes.io/projected/9f5cad2c-4851-4f0f-a04e-c70f832210c6-kube-api-access-4pk4s\") pod \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") "
Nov 02 01:33:21.505940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.505922    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-maas-api-tls\") pod \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") "
Nov 02 01:33:21.505974 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.505958    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5cad2c-4851-4f0f-a04e-c70f832210c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5cad2c-4851-4f0f-a04e-c70f832210c6-openshift-service-ca\") pod \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") "
Nov 02 01:33:21.506011 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.505986    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-metrics-certs\") pod \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\" (UID: \"9f5cad2c-4851-4f0f-a04e-c70f832210c6\") "
Nov 02 01:33:21.506363 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.506337    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5cad2c-4851-4f0f-a04e-c70f832210c6-openshift-service-ca" pod "9f5cad2c-4851-4f0f-a04e-c70f832210c6" (UID: "9f5cad2c-4851-4f0f-a04e-c70f832210c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:33:21.508134 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.508107    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5cad2c-4851-4f0f-a04e-c70f832210c6-kube-api-access-4pk4s" pod "9f5cad2c-4851-4f0f-a04e-c70f832210c6" (UID: "9f5cad2c-4851-4f0f-a04e-c70f832210c6"). InnerVolumeSpecName "kube-api-access-4pk4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:33:21.508230 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.508155    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-maas-api-tls" pod "9f5cad2c-4851-4f0f-a04e-c70f832210c6" (UID: "9f5cad2c-4851-4f0f-a04e-c70f832210c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:21.508276 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.508237    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-metrics-certs" pod "9f5cad2c-4851-4f0f-a04e-c70f832210c6" (UID: "9f5cad2c-4851-4f0f-a04e-c70f832210c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:33:21.607186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.607107    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:21.607186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.607136    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5cad2c-4851-4f0f-a04e-c70f832210c6-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:21.607186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.607149    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f5cad2c-4851-4f0f-a04e-c70f832210c6-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:21.607186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.607158    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pk4s\" (UniqueName: \"kubernetes.io/projected/9f5cad2c-4851-4f0f-a04e-c70f832210c6-kube-api-access-4pk4s\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:33:21.924980 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.924903    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"]
Nov 02 01:33:21.929136 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:21.929109    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7"]
Nov 02 01:33:22.008775 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:22.008737    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-98dt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Nov 02 01:33:22.008940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:22.008795    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Nov 02 01:33:22.236573 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:22.236486    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5cad2c-4851-4f0f-a04e-c70f832210c6" path="/var/lib/kubelet/pods/9f5cad2c-4851-4f0f-a04e-c70f832210c6/volumes"
Nov 02 01:33:23.008383 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:23.008344    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-98dt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Nov 02 01:33:23.008808 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:23.008424    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Nov 02 01:33:24.011598 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:24.011563    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:33:24.011983 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:24.011918    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:33:24.012488 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:24.012470    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:33:24.397810 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:24.397691    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-hq9m7" podUID="9f5cad2c-4851-4f0f-a04e-c70f832210c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.36:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:33:46.037706 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:33:46.037671    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:33:48.480381 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:33:48.480346    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:34:16.037542 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:34:16.037508    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:34:38.017125 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.017035    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"]
Nov 02 01:34:38.017782 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.017557    4287 cpu_manager.go:414] "containerMap: removing container" podUID="9f5cad2c-4851-4f0f-a04e-c70f832210c6" containerName="maas-api"
Nov 02 01:34:38.017782 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.017576    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5cad2c-4851-4f0f-a04e-c70f832210c6" containerName="maas-api"
Nov 02 01:34:38.017782 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.017729    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5cad2c-4851-4f0f-a04e-c70f832210c6" containerName="maas-api"
Nov 02 01:34:38.020624 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.020588    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:38.078892 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.078871    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"]
Nov 02 01:34:38.151421 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.151395    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e0a342-0ab5-4571-86ca-d8da29907012-cert\") pod \"llmisvc-controller-manager-6466c554d8-lqgxq\" (UID: \"62e0a342-0ab5-4571-86ca-d8da29907012\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:38.151557 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.151472    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc2bd\" (UniqueName: \"kubernetes.io/projected/62e0a342-0ab5-4571-86ca-d8da29907012-kube-api-access-bc2bd\") pod \"llmisvc-controller-manager-6466c554d8-lqgxq\" (UID: \"62e0a342-0ab5-4571-86ca-d8da29907012\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:38.252765 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.252735    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc2bd\" (UniqueName: \"kubernetes.io/projected/62e0a342-0ab5-4571-86ca-d8da29907012-kube-api-access-bc2bd\") pod \"llmisvc-controller-manager-6466c554d8-lqgxq\" (UID: \"62e0a342-0ab5-4571-86ca-d8da29907012\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:38.252904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.252785    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e0a342-0ab5-4571-86ca-d8da29907012-cert\") pod \"llmisvc-controller-manager-6466c554d8-lqgxq\" (UID: \"62e0a342-0ab5-4571-86ca-d8da29907012\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:38.255340 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.255317    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62e0a342-0ab5-4571-86ca-d8da29907012-cert\") pod \"llmisvc-controller-manager-6466c554d8-lqgxq\" (UID: \"62e0a342-0ab5-4571-86ca-d8da29907012\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:38.314030 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.313974    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc2bd\" (UniqueName: \"kubernetes.io/projected/62e0a342-0ab5-4571-86ca-d8da29907012-kube-api-access-bc2bd\") pod \"llmisvc-controller-manager-6466c554d8-lqgxq\" (UID: \"62e0a342-0ab5-4571-86ca-d8da29907012\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:38.332305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.332282    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:38.702175 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.702151    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"]
Nov 02 01:34:38.704182 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:34:38.704150    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e0a342_0ab5_4571_86ca_d8da29907012.slice/crio-54deaef774499094ff33b883868813782e373c140e65a141714ca9bde98a29cc WatchSource:0}: Error finding container 54deaef774499094ff33b883868813782e373c140e65a141714ca9bde98a29cc: Status 404 returned error can't find the container with id 54deaef774499094ff33b883868813782e373c140e65a141714ca9bde98a29cc
Nov 02 01:34:38.852217 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:38.852181    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq" event={"ID":"62e0a342-0ab5-4571-86ca-d8da29907012","Type":"ContainerStarted","Data":"54deaef774499094ff33b883868813782e373c140e65a141714ca9bde98a29cc"}
Nov 02 01:34:39.858417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:39.858386    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq" event={"ID":"62e0a342-0ab5-4571-86ca-d8da29907012","Type":"ContainerStarted","Data":"969e1f80265aa9542618cc43fa8cb94898bba918a641c8eee07a519f667efc52"}
Nov 02 01:34:39.858807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:39.858545    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:34:39.975973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:39.975920    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq" podStartSLOduration=2.195997348 podStartE2EDuration="2.975906297s" totalImagesPullingTime="779.908949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:34:39.975034673 +0000 UTC m=+587.848675725" watchObservedRunningTime="2026-11-02 01:34:39.975906297 +0000 UTC m=+587.849547350"
Nov 02 01:34:46.037665 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:34:46.037626    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:34:52.264803 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:52.264774    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:34:52.266142 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:34:52.266121    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:35:10.865364 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:10.865334    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqgxq"
Nov 02 01:35:11.033143 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.033100    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"]
Nov 02 01:35:11.033377 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.033349    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9" podUID="80d48db7-565a-4d7a-92b5-1825b3ba74ce" containerName="manager" containerID="cri-o://8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c" gracePeriod=30
Nov 02 01:35:11.279158 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.279134    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:35:11.323619 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.323588    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80d48db7-565a-4d7a-92b5-1825b3ba74ce-cert\" (UniqueName: \"kubernetes.io/secret/80d48db7-565a-4d7a-92b5-1825b3ba74ce-cert\") pod \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\" (UID: \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\") "
Nov 02 01:35:11.323742 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.323688    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d48db7-565a-4d7a-92b5-1825b3ba74ce-kube-api-access-6rqsl\" (UniqueName: \"kubernetes.io/projected/80d48db7-565a-4d7a-92b5-1825b3ba74ce-kube-api-access-6rqsl\") pod \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\" (UID: \"80d48db7-565a-4d7a-92b5-1825b3ba74ce\") "
Nov 02 01:35:11.325696 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.325671    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d48db7-565a-4d7a-92b5-1825b3ba74ce-cert" pod "80d48db7-565a-4d7a-92b5-1825b3ba74ce" (UID: "80d48db7-565a-4d7a-92b5-1825b3ba74ce"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:35:11.325817 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.325796    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d48db7-565a-4d7a-92b5-1825b3ba74ce-kube-api-access-6rqsl" pod "80d48db7-565a-4d7a-92b5-1825b3ba74ce" (UID: "80d48db7-565a-4d7a-92b5-1825b3ba74ce"). InnerVolumeSpecName "kube-api-access-6rqsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:35:11.424473 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.424409    4287 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80d48db7-565a-4d7a-92b5-1825b3ba74ce-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:35:11.424473 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:11.424432    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqsl\" (UniqueName: \"kubernetes.io/projected/80d48db7-565a-4d7a-92b5-1825b3ba74ce-kube-api-access-6rqsl\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:35:12.016702 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.016666    4287 generic.go:356] "Generic (PLEG): container finished" podID="80d48db7-565a-4d7a-92b5-1825b3ba74ce" containerID="8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c" exitCode=0
Nov 02 01:35:12.017186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.016709    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"
Nov 02 01:35:12.017186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.016749    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9" event={"ID":"80d48db7-565a-4d7a-92b5-1825b3ba74ce","Type":"ContainerDied","Data":"8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c"}
Nov 02 01:35:12.017186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.016800    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9" event={"ID":"80d48db7-565a-4d7a-92b5-1825b3ba74ce","Type":"ContainerDied","Data":"e3e01ccbb7f0553aee2ebab4313ea6b54c9c6de9ee69831579d9293e0f698c07"}
Nov 02 01:35:12.017186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.016821    4287 scope.go:122] "RemoveContainer" containerID="8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c"
Nov 02 01:35:12.025774 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.025755    4287 scope.go:122] "RemoveContainer" containerID="8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c"
Nov 02 01:35:12.026042 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:35:12.026023    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c\": container with ID starting with 8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c not found: ID does not exist" containerID="8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c"
Nov 02 01:35:12.026105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.026054    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c"} err="failed to get container status \"8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c\": rpc error: code = NotFound desc = could not find container \"8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c\": container with ID starting with 8979633bdd35dbcf055e72a62d70b3dec6dd2c58a2335c5230329eae808b171c not found: ID does not exist"
Nov 02 01:35:12.183446 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.183412    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"]
Nov 02 01:35:12.237180 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:12.237157    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tr9r9"]
Nov 02 01:35:14.235623 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:35:14.235588    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d48db7-565a-4d7a-92b5-1825b3ba74ce" path="/var/lib/kubelet/pods/80d48db7-565a-4d7a-92b5-1825b3ba74ce/volumes"
Nov 02 01:35:16.039622 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:35:16.039573    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:35:46.038370 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:35:46.038332    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:36:08.593795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.593760    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"]
Nov 02 01:36:08.594223 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.594118    4287 cpu_manager.go:414] "containerMap: removing container" podUID="80d48db7-565a-4d7a-92b5-1825b3ba74ce" containerName="manager"
Nov 02 01:36:08.594223 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.594129    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d48db7-565a-4d7a-92b5-1825b3ba74ce" containerName="manager"
Nov 02 01:36:08.594223 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.594188    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d48db7-565a-4d7a-92b5-1825b3ba74ce" containerName="manager"
Nov 02 01:36:08.597235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.597218    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.691803 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.691754    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.691970 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.691807    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.691970 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.691901    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.691970 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.691949    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.692092 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.692062    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqfl\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-kube-api-access-dvqfl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.692144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.692112    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a81fde-83aa-47ba-95da-64721bc396e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.692248 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.692229    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.692314 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.692295    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a81fde-83aa-47ba-95da-64721bc396e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.692365 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.692334    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.699393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.699364    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"]
Nov 02 01:36:08.792713 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792673    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a81fde-83aa-47ba-95da-64721bc396e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.792713 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792718    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.792987 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792779    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a81fde-83aa-47ba-95da-64721bc396e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.792987 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792814    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.792987 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792864    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.792987 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792893    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.792987 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792918    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.792987 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792947    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.793354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.792991    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqfl\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-kube-api-access-dvqfl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.793354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.793233    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.793482 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.793416    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.793660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.793535    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.793762 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.793693    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.793854 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.793834    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a81fde-83aa-47ba-95da-64721bc396e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.795351 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.795329    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.795514 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.795491    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a81fde-83aa-47ba-95da-64721bc396e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.885041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.884963    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.892092 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.892065    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqfl\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-kube-api-access-dvqfl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-b8wvj\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:08.908908 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:08.908883    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:09.336873 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:09.336846    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"]
Nov 02 01:36:09.339510 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:36:09.339481    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a81fde_83aa_47ba_95da_64721bc396e0.slice/crio-e56d1d37e3ff4b65b1c39ff2df096b16b246761bbb181ff496967b28eab162ff WatchSource:0}: Error finding container e56d1d37e3ff4b65b1c39ff2df096b16b246761bbb181ff496967b28eab162ff: Status 404 returned error can't find the container with id e56d1d37e3ff4b65b1c39ff2df096b16b246761bbb181ff496967b28eab162ff
Nov 02 01:36:09.342021 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:09.341985    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:09.342095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:09.342071    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:09.342143 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:09.342114    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:36:10.316359 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:10.316311    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" event={"ID":"77a81fde-83aa-47ba-95da-64721bc396e0","Type":"ContainerStarted","Data":"170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7"}
Nov 02 01:36:10.316975 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:10.316385    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" event={"ID":"77a81fde-83aa-47ba-95da-64721bc396e0","Type":"ContainerStarted","Data":"e56d1d37e3ff4b65b1c39ff2df096b16b246761bbb181ff496967b28eab162ff"}
Nov 02 01:36:10.440406 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:10.440347    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" podStartSLOduration=2.440332887 podStartE2EDuration="2.440332887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:10.439393167 +0000 UTC m=+678.313034218" watchObservedRunningTime="2026-11-02 01:36:10.440332887 +0000 UTC m=+678.313973939"
Nov 02 01:36:10.909513 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:10.909480    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:10.911057 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:10.911026    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 01:36:10.911182 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:10.911095    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 01:36:11.909286 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:11.909249    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 01:36:11.909704 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:11.909308    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 01:36:12.909794 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:12.909758    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 01:36:12.910210 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:12.909814    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 01:36:13.909453 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:13.909413    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 01:36:13.909660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:13.909471    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 01:36:14.910099 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:14.910063    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 01:36:14.910501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:14.910127    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 01:36:15.909583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:15.909546    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 01:36:15.909800 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:15.909628    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 01:36:16.037715 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:36:16.037674    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:36:17.563115 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:17.563079    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:17.563739 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:17.563470    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:17.564336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:36:17.564312    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:36:46.038164 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:36:46.038124    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:37:16.038569 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:37:16.038534    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:37:46.037839 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:37:46.037802    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:37:50.531903 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.531861    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"]
Nov 02 01:37:50.535361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.535340    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.573764 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.573734    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.573862 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.573768    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-metrics-certs\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.573901 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.573861    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7fpd\" (UniqueName: \"kubernetes.io/projected/e7fe6eac-4558-4b31-bf54-c7b18365f915-kube-api-access-n7fpd\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.573936 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.573923    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fe6eac-4558-4b31-bf54-c7b18365f915-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.587258 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.587238    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-e06ea2-serving-cert\""
Nov 02 01:37:50.638932 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.638899    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"]
Nov 02 01:37:50.674695 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.674672    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fe6eac-4558-4b31-bf54-c7b18365f915-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.674807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.674701    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.674807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.674719    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-metrics-certs\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.674807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.674767    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7fpd\" (UniqueName: \"kubernetes.io/projected/e7fe6eac-4558-4b31-bf54-c7b18365f915-kube-api-access-n7fpd\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.674973 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:37:50.674824    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-serving-cert: secret "maas-api-e2e-worker-w5-e06ea2-serving-cert" not found
Nov 02 01:37:50.674973 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:37:50.674900    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls podName:e7fe6eac-4558-4b31-bf54-c7b18365f915 nodeName:}" failed. No retries permitted until 2026-11-02 01:37:51.174882091 +0000 UTC m=+779.048523123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls") pod "maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55" (UID: "e7fe6eac-4558-4b31-bf54-c7b18365f915") : secret "maas-api-e2e-worker-w5-e06ea2-serving-cert" not found
Nov 02 01:37:50.675358 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.675336    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fe6eac-4558-4b31-bf54-c7b18365f915-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.677198 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.677180    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-metrics-certs\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:50.735540 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:50.735517    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7fpd\" (UniqueName: \"kubernetes.io/projected/e7fe6eac-4558-4b31-bf54-c7b18365f915-kube-api-access-n7fpd\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:51.178161 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:51.178130    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:51.180464 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:51.180438    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:51.447579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:51.447507    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:51.818254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:51.818225    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"]
Nov 02 01:37:51.819951 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:37:51.819923    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fe6eac_4558_4b31_bf54_c7b18365f915.slice/crio-2e2930810f0b4bdb0e6c7a8c76cda94911bfc958d0d065f47d4f36a0d370ae7b WatchSource:0}: Error finding container 2e2930810f0b4bdb0e6c7a8c76cda94911bfc958d0d065f47d4f36a0d370ae7b: Status 404 returned error can't find the container with id 2e2930810f0b4bdb0e6c7a8c76cda94911bfc958d0d065f47d4f36a0d370ae7b
Nov 02 01:37:51.821193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:51.821176    4287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:37:52.812370 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:52.812320    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55" event={"ID":"e7fe6eac-4558-4b31-bf54-c7b18365f915","Type":"ContainerStarted","Data":"562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f"}
Nov 02 01:37:52.812550 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:52.812390    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55" event={"ID":"e7fe6eac-4558-4b31-bf54-c7b18365f915","Type":"ContainerStarted","Data":"2e2930810f0b4bdb0e6c7a8c76cda94911bfc958d0d065f47d4f36a0d370ae7b"}
Nov 02 01:37:52.812550 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:52.812471    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:52.968963 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:52.968860    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55" podStartSLOduration=2.103030394 podStartE2EDuration="2.96884706s" totalImagesPullingTime="865.816666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:52.967298944 +0000 UTC m=+780.840939997" watchObservedRunningTime="2026-11-02 01:37:52.96884706 +0000 UTC m=+780.842488111"
Nov 02 01:37:55.821222 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:55.821184    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"]
Nov 02 01:37:55.824548 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:55.824526    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:55.877503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:55.877467    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-36a1e0-serving-cert\""
Nov 02 01:37:55.918142 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:55.918114    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-metrics-certs\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:55.918291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:55.918155    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:55.918291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:55.918177    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e4fa5c-9232-42d1-b6ea-cf9448c24602-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:55.918413 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:55.918328    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qq2d\" (UniqueName: \"kubernetes.io/projected/12e4fa5c-9232-42d1-b6ea-cf9448c24602-kube-api-access-8qq2d\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:55.929082 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:55.929053    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"]
Nov 02 01:37:56.019126 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.019082    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qq2d\" (UniqueName: \"kubernetes.io/projected/12e4fa5c-9232-42d1-b6ea-cf9448c24602-kube-api-access-8qq2d\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.019334 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.019135    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-metrics-certs\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.019334 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.019173    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.019334 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.019205    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e4fa5c-9232-42d1-b6ea-cf9448c24602-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.019334 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:37:56.019307    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-serving-cert: secret "maas-api-e2e-worker-w2-36a1e0-serving-cert" not found
Nov 02 01:37:56.019561 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:37:56.019381    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls podName:12e4fa5c-9232-42d1-b6ea-cf9448c24602 nodeName:}" failed. No retries permitted until 2026-11-02 01:37:56.519364722 +0000 UTC m=+784.393005754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls") pod "maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d" (UID: "12e4fa5c-9232-42d1-b6ea-cf9448c24602") : secret "maas-api-e2e-worker-w2-36a1e0-serving-cert" not found
Nov 02 01:37:56.019962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.019940    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e4fa5c-9232-42d1-b6ea-cf9448c24602-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.021944 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.021921    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-metrics-certs\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.080531 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.080460    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qq2d\" (UniqueName: \"kubernetes.io/projected/12e4fa5c-9232-42d1-b6ea-cf9448c24602-kube-api-access-8qq2d\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.524110 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.524084    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.526699 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.526676    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls\") pod \"maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:56.742557 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:56.742510    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:57.116693 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:57.116668    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"]
Nov 02 01:37:57.844556 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:57.844518    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d" event={"ID":"12e4fa5c-9232-42d1-b6ea-cf9448c24602","Type":"ContainerStarted","Data":"f1c31389603e8a4e0d08dfef3c531b03eff742361a5ce9e796d98a68f5082356"}
Nov 02 01:37:58.823240 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:58.823211    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 01:37:58.853777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:58.853738    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d" event={"ID":"12e4fa5c-9232-42d1-b6ea-cf9448c24602","Type":"ContainerStarted","Data":"2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e"}
Nov 02 01:37:58.853939 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:58.853881    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:37:59.059179 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:37:59.059117    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d" podStartSLOduration=3.177643843 podStartE2EDuration="4.059099911s" totalImagesPullingTime="881.456068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:59.057065957 +0000 UTC m=+786.930707010" watchObservedRunningTime="2026-11-02 01:37:59.059099911 +0000 UTC m=+786.932740962"
Nov 02 01:38:00.716042 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.716009    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"]
Nov 02 01:38:00.719581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.719559    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.761448 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.761418    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-metrics-certs\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.761592 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.761474    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhw67\" (UniqueName: \"kubernetes.io/projected/966d6023-b96f-4e93-8b05-f849e473b4fe-kube-api-access-hhw67\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.761741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.761702    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966d6023-b96f-4e93-8b05-f849e473b4fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.761869 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.761846    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.771336 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.771317    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-f276c3-serving-cert\""
Nov 02 01:38:00.823283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.823249    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"]
Nov 02 01:38:00.862429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.862392    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.862785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.862492    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-metrics-certs\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.862785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.862533    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhw67\" (UniqueName: \"kubernetes.io/projected/966d6023-b96f-4e93-8b05-f849e473b4fe-kube-api-access-hhw67\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.862785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.862595    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966d6023-b96f-4e93-8b05-f849e473b4fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.863396 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.863370    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966d6023-b96f-4e93-8b05-f849e473b4fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.863506 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:38:00.863491    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-serving-cert: secret "maas-api-e2e-worker-w1-f276c3-serving-cert" not found
Nov 02 01:38:00.863564 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:38:00.863551    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls podName:966d6023-b96f-4e93-8b05-f849e473b4fe nodeName:}" failed. No retries permitted until 2026-11-02 01:38:01.363532659 +0000 UTC m=+789.237173695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls") pod "maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp" (UID: "966d6023-b96f-4e93-8b05-f849e473b4fe") : secret "maas-api-e2e-worker-w1-f276c3-serving-cert" not found
Nov 02 01:38:00.866247 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.866224    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-metrics-certs\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:00.924278 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:00.924250    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhw67\" (UniqueName: \"kubernetes.io/projected/966d6023-b96f-4e93-8b05-f849e473b4fe-kube-api-access-hhw67\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:01.367202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:01.367160    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:01.371374 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:01.371346    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls\") pod \"maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:01.631708 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:01.631628    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:02.006723 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:02.006693    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"]
Nov 02 01:38:02.008035 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:38:02.008006    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966d6023_b96f_4e93_8b05_f849e473b4fe.slice/crio-7b4ebf77203518f46e088ffc3d479a18913c0656408e3c271505d81769f62c3c WatchSource:0}: Error finding container 7b4ebf77203518f46e088ffc3d479a18913c0656408e3c271505d81769f62c3c: Status 404 returned error can't find the container with id 7b4ebf77203518f46e088ffc3d479a18913c0656408e3c271505d81769f62c3c
Nov 02 01:38:02.876732 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:02.876699    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp" event={"ID":"966d6023-b96f-4e93-8b05-f849e473b4fe","Type":"ContainerStarted","Data":"7b4ebf77203518f46e088ffc3d479a18913c0656408e3c271505d81769f62c3c"}
Nov 02 01:38:03.884324 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:03.884290    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp" event={"ID":"966d6023-b96f-4e93-8b05-f849e473b4fe","Type":"ContainerStarted","Data":"86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e"}
Nov 02 01:38:03.884882 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:03.884406    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:04.003095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:04.003038    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp" podStartSLOduration=3.197844824 podStartE2EDuration="4.003026107s" totalImagesPullingTime="805.181283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:04.000838377 +0000 UTC m=+791.874479428" watchObservedRunningTime="2026-11-02 01:38:04.003026107 +0000 UTC m=+791.876667159"
Nov 02 01:38:04.864464 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:04.864436    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 01:38:05.622266 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.622231    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"]
Nov 02 01:38:05.625813 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.625797    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.677755 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.677729    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-88e76d-serving-cert\""
Nov 02 01:38:05.684650 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.684625    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"]
Nov 02 01:38:05.711068 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.711043    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptp6x\" (UniqueName: \"kubernetes.io/projected/1f0d821c-ae92-41fa-975d-940af48c0730-kube-api-access-ptp6x\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.711165 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.711150    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.711206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.711171    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-metrics-certs\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.711260 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.711200    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d821c-ae92-41fa-975d-940af48c0730-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.812103 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.812074    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptp6x\" (UniqueName: \"kubernetes.io/projected/1f0d821c-ae92-41fa-975d-940af48c0730-kube-api-access-ptp6x\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.812256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.812145    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.812256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.812161    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-metrics-certs\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.812256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.812180    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d821c-ae92-41fa-975d-940af48c0730-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.812417 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:38:05.812316    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-serving-cert: secret "maas-api-e2e-worker-w3-88e76d-serving-cert" not found
Nov 02 01:38:05.812417 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:38:05.812389    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls podName:1f0d821c-ae92-41fa-975d-940af48c0730 nodeName:}" failed. No retries permitted until 2026-11-02 01:38:06.312367778 +0000 UTC m=+794.186008822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls") pod "maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv" (UID: "1f0d821c-ae92-41fa-975d-940af48c0730") : secret "maas-api-e2e-worker-w3-88e76d-serving-cert" not found
Nov 02 01:38:05.812747 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.812728    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d821c-ae92-41fa-975d-940af48c0730-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.814762 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.814740    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-metrics-certs\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:05.873255 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:05.873191    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptp6x\" (UniqueName: \"kubernetes.io/projected/1f0d821c-ae92-41fa-975d-940af48c0730-kube-api-access-ptp6x\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:06.316181 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:06.316151    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:06.318579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:06.318552    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls\") pod \"maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:06.538655 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:06.538601    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:06.910455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:06.910433    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"]
Nov 02 01:38:06.911892 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:38:06.911868    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0d821c_ae92_41fa_975d_940af48c0730.slice/crio-5943ce98fbad1b505d5eff96622fd7ba54424f0d9d9a2275be6cd5ed296b5db3 WatchSource:0}: Error finding container 5943ce98fbad1b505d5eff96622fd7ba54424f0d9d9a2275be6cd5ed296b5db3: Status 404 returned error can't find the container with id 5943ce98fbad1b505d5eff96622fd7ba54424f0d9d9a2275be6cd5ed296b5db3
Nov 02 01:38:07.911661 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:07.911633    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv" event={"ID":"1f0d821c-ae92-41fa-975d-940af48c0730","Type":"ContainerStarted","Data":"238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a"}
Nov 02 01:38:07.912023 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:07.911680    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv" event={"ID":"1f0d821c-ae92-41fa-975d-940af48c0730","Type":"ContainerStarted","Data":"5943ce98fbad1b505d5eff96622fd7ba54424f0d9d9a2275be6cd5ed296b5db3"}
Nov 02 01:38:07.912023 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:07.911725    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:08.061899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:08.061774    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv" podStartSLOduration=2.193456093 podStartE2EDuration="3.061752376s" totalImagesPullingTime="868.296283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:08.058453007 +0000 UTC m=+795.932094060" watchObservedRunningTime="2026-11-02 01:38:08.061752376 +0000 UTC m=+795.935393430"
Nov 02 01:38:09.893992 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:09.893965    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 01:38:13.922835 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:13.922800    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 01:38:16.037866 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:38:16.037832    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:38:20.522123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.522091    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"]
Nov 02 01:38:20.525511 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.525494    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.577535 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.577511    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-187478-serving-cert\""
Nov 02 01:38:20.629040 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.629012    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"]
Nov 02 01:38:20.644540 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.644515    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxd4\" (UniqueName: \"kubernetes.io/projected/048d59f4-304c-4e3f-a645-05475489a0da-kube-api-access-hwxd4\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.644682 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.644557    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.644682 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.644580    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048d59f4-304c-4e3f-a645-05475489a0da-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.644772 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.644729    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-metrics-certs\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.745807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.745770    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.745974 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.745813    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048d59f4-304c-4e3f-a645-05475489a0da-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.745974 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:38:20.745936    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-serving-cert: secret "maas-api-e2e-worker-w4-187478-serving-cert" not found
Nov 02 01:38:20.746096 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:38:20.746002    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls podName:048d59f4-304c-4e3f-a645-05475489a0da nodeName:}" failed. No retries permitted until 2026-11-02 01:38:21.245981494 +0000 UTC m=+809.119622524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls") pod "maas-api-e2e-worker-w4-187478-845f545876-cvqhj" (UID: "048d59f4-304c-4e3f-a645-05475489a0da") : secret "maas-api-e2e-worker-w4-187478-serving-cert" not found
Nov 02 01:38:20.746096 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.746025    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-metrics-certs\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.746202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.746127    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxd4\" (UniqueName: \"kubernetes.io/projected/048d59f4-304c-4e3f-a645-05475489a0da-kube-api-access-hwxd4\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.746529 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.746509    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048d59f4-304c-4e3f-a645-05475489a0da-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.748670 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.748650    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-metrics-certs\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:20.809837 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:20.809733    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxd4\" (UniqueName: \"kubernetes.io/projected/048d59f4-304c-4e3f-a645-05475489a0da-kube-api-access-hwxd4\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:21.249801 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:21.249724    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:21.252163 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:21.252138    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls\") pod \"maas-api-e2e-worker-w4-187478-845f545876-cvqhj\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:21.437403 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:21.437371    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:21.810268 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:21.810244    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"]
Nov 02 01:38:21.812150 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:38:21.812110    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048d59f4_304c_4e3f_a645_05475489a0da.slice/crio-d26c29977c5e255db8e0a2f4df9fec88dfc8addbb2564039879206ebd220b4dd WatchSource:0}: Error finding container d26c29977c5e255db8e0a2f4df9fec88dfc8addbb2564039879206ebd220b4dd: Status 404 returned error can't find the container with id d26c29977c5e255db8e0a2f4df9fec88dfc8addbb2564039879206ebd220b4dd
Nov 02 01:38:21.980781 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:21.980747    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj" event={"ID":"048d59f4-304c-4e3f-a645-05475489a0da","Type":"ContainerStarted","Data":"d26c29977c5e255db8e0a2f4df9fec88dfc8addbb2564039879206ebd220b4dd"}
Nov 02 01:38:22.988208 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:22.988112    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj" event={"ID":"048d59f4-304c-4e3f-a645-05475489a0da","Type":"ContainerStarted","Data":"4e682b6cce0479b09b64f227491f15a8f3f0bdb16b11d52c945b28181986e087"}
Nov 02 01:38:22.988576 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:22.988210    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:23.105227 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:23.105175    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj" podStartSLOduration=2.292786604 podStartE2EDuration="3.105162583s" totalImagesPullingTime="812.375979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:23.104546021 +0000 UTC m=+810.978187074" watchObservedRunningTime="2026-11-02 01:38:23.105162583 +0000 UTC m=+810.978803634"
Nov 02 01:38:28.998583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:38:28.998545    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 01:38:46.038396 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:38:46.038357    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:39:16.038526 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:39:16.038486    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:39:46.038952 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:39:46.038918    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:39:52.306385 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:39:52.306354    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:39:52.309822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:39:52.309802    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:40:16.038434 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:40:16.038400    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:40:19.227315 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.227274    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"]
Nov 02 01:40:19.230972 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.230953    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.332545 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.332504    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"kube-root-ca.crt\""
Nov 02 01:40:19.332743 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.332570    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"premium-w2-36a1e0-kserve-self-signed-certs\""
Nov 02 01:40:19.332743 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.332559    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"openshift-service-ca.crt\""
Nov 02 01:40:19.332743 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.332573    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"default-dockercfg-t2p55\""
Nov 02 01:40:19.334968 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.334946    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"]
Nov 02 01:40:19.395394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.395356    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6w5\" (UniqueName: \"kubernetes.io/projected/19694632-5293-4f90-bd57-e118aa208699-kube-api-access-9m6w5\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.395637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.395480    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-home\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.395637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.395570    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-model-cache\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.395637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.395601    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-tmp-dir\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.395797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.395659    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19694632-5293-4f90-bd57-e118aa208699-tls-certs\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.395797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.395701    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-dshm\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.496592 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.496510    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-home\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.496777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.496590    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-model-cache\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.496777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.496645    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-tmp-dir\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.496777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.496679    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19694632-5293-4f90-bd57-e118aa208699-tls-certs\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.496777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.496711    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-dshm\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.496777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.496764    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6w5\" (UniqueName: \"kubernetes.io/projected/19694632-5293-4f90-bd57-e118aa208699-kube-api-access-9m6w5\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.497034 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.496985    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-model-cache\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.497034 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.497003    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-home\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.497260 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.497234    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-tmp-dir\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.499285 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.499255    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-dshm\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.499579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.499560    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19694632-5293-4f90-bd57-e118aa208699-tls-certs\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.557663 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.557627    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6w5\" (UniqueName: \"kubernetes.io/projected/19694632-5293-4f90-bd57-e118aa208699-kube-api-access-9m6w5\") pod \"premium-w2-36a1e0-kserve-6d56bcf654-bzv9n\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") " pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:19.842464 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:19.842430    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:20.218268 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:20.218241    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"]
Nov 02 01:40:20.220213 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:40:20.220185    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19694632_5293_4f90_bd57_e118aa208699.slice/crio-2e7c55beb45d721085f873f89cda3d4fca5838f50e35ef0afa5e436da0b0f8b4 WatchSource:0}: Error finding container 2e7c55beb45d721085f873f89cda3d4fca5838f50e35ef0afa5e436da0b0f8b4: Status 404 returned error can't find the container with id 2e7c55beb45d721085f873f89cda3d4fca5838f50e35ef0afa5e436da0b0f8b4
Nov 02 01:40:20.567712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:20.567678    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" event={"ID":"19694632-5293-4f90-bd57-e118aa208699","Type":"ContainerStarted","Data":"2e7c55beb45d721085f873f89cda3d4fca5838f50e35ef0afa5e436da0b0f8b4"}
Nov 02 01:40:23.587169 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:23.587128    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" event={"ID":"19694632-5293-4f90-bd57-e118aa208699","Type":"ContainerStarted","Data":"9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098"}
Nov 02 01:40:23.705505 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:23.705450    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podStartSLOduration=2.32534465 podStartE2EDuration="4.7054355s" totalImagesPullingTime="2.38009085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:23.703994284 +0000 UTC m=+931.577635337" watchObservedRunningTime="2026-11-02 01:40:23.7054355 +0000 UTC m=+931.579076551"
Nov 02 01:40:23.843074 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:23.842993    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:29.842887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:29.842845    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:29.855325 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:29.855297    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:30.632307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:30.632274    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:40:37.194870 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.194781    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"]
Nov 02 01:40:37.199540 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.199514    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.251670 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.251633    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e06ea2-openshift-default-dockercfg-x8mvt\""
Nov 02 01:40:37.252061 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252040    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.252113 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252072    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.252113 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252090    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.252227 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252209    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70414325-f032-436c-9bb1-951077b61520-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.252278 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252260    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.252388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252369    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70414325-f032-436c-9bb1-951077b61520-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.252450 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252407    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5lsc\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-kube-api-access-t5lsc\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.252510 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252477    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.252565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.252543    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.304671 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.304596    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"]
Nov 02 01:40:37.353688 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.353645    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.353899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.353699    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.353899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.353753    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.353899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.353770    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.353952    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354046    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70414325-f032-436c-9bb1-951077b61520-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354100    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354122    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354126    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354183    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70414325-f032-436c-9bb1-951077b61520-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354202    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354231    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5lsc\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-kube-api-access-t5lsc\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354554 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354417    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.354816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.354792    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70414325-f032-436c-9bb1-951077b61520-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.356416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.356392    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.356651 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.356601    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70414325-f032-436c-9bb1-951077b61520-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.414553 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.414524    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.414713 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.414601    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5lsc\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-kube-api-access-t5lsc\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.512347 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.512300    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:37.796864 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.796756    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"]
Nov 02 01:40:37.801876 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.801851    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.854098 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.854060    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-36a1e0-openshift-default-dockercfg-lck7x\""
Nov 02 01:40:37.858432 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.858398    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx55q\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-kube-api-access-wx55q\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.858876 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.858828    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.858876 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.858861    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.859048 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.858886    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.859093 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.859047    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.859202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.859184    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.859246 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.859221    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.859290 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.859248    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.859338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.859291    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3b16cf-2480-4289-ab1a-e429adb66645-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.861585 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.861560    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"]
Nov 02 01:40:37.941578 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.941552    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"]
Nov 02 01:40:37.943457 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:40:37.943430    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70414325_f032_436c_9bb1_951077b61520.slice/crio-d98f4a1585f9e38eb0478bf840cf59ba77064f3bb58b3e238220c04adb850c23 WatchSource:0}: Error finding container d98f4a1585f9e38eb0478bf840cf59ba77064f3bb58b3e238220c04adb850c23: Status 404 returned error can't find the container with id d98f4a1585f9e38eb0478bf840cf59ba77064f3bb58b3e238220c04adb850c23
Nov 02 01:40:37.945857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.945815    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:37.945952 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.945892    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:37.945952 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.945936    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:37.960109 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960085    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960199 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960119    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960199 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960142    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960199 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960183    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960258    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960289    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960321    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960350    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3b16cf-2480-4289-ab1a-e429adb66645-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960397    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx55q\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-kube-api-access-wx55q\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.960864 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.960813    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.961060 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.961021    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.961194 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.961176    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3b16cf-2480-4289-ab1a-e429adb66645-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.961289 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.961193    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.961289 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.961275    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.962656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.962635    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:37.962926 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:37.962906    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:38.022171 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.022089    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx55q\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-kube-api-access-wx55q\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:38.022171 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.022161    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:38.115286 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.115192    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:38.547722 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.547692    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"]
Nov 02 01:40:38.575177 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.573357    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:38.575177 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.573455    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:38.575177 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.573497    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:40:38.679645 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.679593    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" event={"ID":"bd3b16cf-2480-4289-ab1a-e429adb66645","Type":"ContainerStarted","Data":"bb667df5b68dfaabf9730ea4679af1480670ee1ad0a1b6d854144c410d527fd4"}
Nov 02 01:40:38.681358 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.681329    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" event={"ID":"70414325-f032-436c-9bb1-951077b61520","Type":"ContainerStarted","Data":"8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2"}
Nov 02 01:40:38.681486 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:38.681377    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" event={"ID":"70414325-f032-436c-9bb1-951077b61520","Type":"ContainerStarted","Data":"d98f4a1585f9e38eb0478bf840cf59ba77064f3bb58b3e238220c04adb850c23"}
Nov 02 01:40:39.513462 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:39.513425    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:39.515020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:39.514990    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 02 01:40:39.515126 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:39.515040    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 02 01:40:39.704816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:39.704771    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" event={"ID":"bd3b16cf-2480-4289-ab1a-e429adb66645","Type":"ContainerStarted","Data":"c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f"}
Nov 02 01:40:39.829345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:39.829218    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" podStartSLOduration=2.829199085 podStartE2EDuration="2.829199085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:39.828407911 +0000 UTC m=+947.702048963" watchObservedRunningTime="2026-11-02 01:40:39.829199085 +0000 UTC m=+947.702840138"
Nov 02 01:40:39.830388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:39.830353    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podStartSLOduration=2.83034178 podStartE2EDuration="2.83034178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:38.801162515 +0000 UTC m=+946.674803567" watchObservedRunningTime="2026-11-02 01:40:39.83034178 +0000 UTC m=+947.703982829"
Nov 02 01:40:40.115640 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:40.115539    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:40.116931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:40.116909    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 01:40:40.116989 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:40.116963    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 01:40:40.512775 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:40.512741    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 02 01:40:40.512924 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:40.512806    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 02 01:40:41.116063 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:41.116026    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 01:40:41.116675 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:41.116640    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 01:40:41.513051 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:41.513000    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 02 01:40:41.513249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:41.513073    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 02 01:40:42.116424 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:42.116388    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 01:40:42.116873 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:42.116437    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 01:40:42.513352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:42.513321    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 02 01:40:42.513515 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:42.513374    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 02 01:40:43.116711 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:43.116156    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 01:40:43.116711 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:43.116221    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 01:40:43.513034 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:43.512982    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 02 01:40:43.513238 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:43.513046    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 02 01:40:44.116015 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:44.115982    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 01:40:44.116238 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:44.116044    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 01:40:44.512949 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:44.512917    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Nov 02 01:40:44.513371 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:44.512986    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Nov 02 01:40:45.115962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:45.115917    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 02 01:40:45.116186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:45.115999    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 02 01:40:45.517542 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:45.517508    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:45.517938 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:45.517868    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:45.518714 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:45.518694    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 01:40:46.039154 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:40:46.039114    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:40:46.120364 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:46.120329    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:46.120736 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:46.120715    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:40:46.121443 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:40:46.121428    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 01:41:16.041350 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:41:16.041269    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:41:46.038251 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:41:46.038214    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:42:07.570801 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.570762    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"]
Nov 02 01:42:07.575272 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.575240    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.648965 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.648926    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.649141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.648977    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.649141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.649094    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.649141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.649121    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c19939-a073-48f8-af1e-d5695cf3327f-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.649252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.649149    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.649252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.649231    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.649322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.649258    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.649322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.649275    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfpc\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-kube-api-access-krfpc\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.649322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.649310    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.677359 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.677334    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5888dc-openshift-default-dockercfg-cgmjh\""
Nov 02 01:42:07.680025 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.680003    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"]
Nov 02 01:42:07.749812 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.749783    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750012 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.749822    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750012 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.749969    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750020    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c19939-a073-48f8-af1e-d5695cf3327f-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750072    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750134    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750166    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750191    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfpc\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-kube-api-access-krfpc\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750377 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750237    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750377 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750244    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750652 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750596    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750754 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750676    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750754 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750721    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.750850 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.750782    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c19939-a073-48f8-af1e-d5695cf3327f-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.752314 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.752295    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.752818 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.752798    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.814912 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.814844    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfpc\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-kube-api-access-krfpc\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.819379 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.819349    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:07.890663 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:07.890561    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:08.316601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:08.316576    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"]
Nov 02 01:42:08.317683 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:42:08.317654    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c19939_a073_48f8_af1e_d5695cf3327f.slice/crio-eb942eab81461ccf0ad2313aaa7a7e67c73ea14f163c4d09f3835b94799d2458 WatchSource:0}: Error finding container eb942eab81461ccf0ad2313aaa7a7e67c73ea14f163c4d09f3835b94799d2458: Status 404 returned error can't find the container with id eb942eab81461ccf0ad2313aaa7a7e67c73ea14f163c4d09f3835b94799d2458
Nov 02 01:42:08.320122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:08.320087    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:08.320207 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:08.320166    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:08.320207 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:08.320197    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:09.211345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:09.211305    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" event={"ID":"d5c19939-a073-48f8-af1e-d5695cf3327f","Type":"ContainerStarted","Data":"9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950"}
Nov 02 01:42:09.211866 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:09.211364    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" event={"ID":"d5c19939-a073-48f8-af1e-d5695cf3327f","Type":"ContainerStarted","Data":"eb942eab81461ccf0ad2313aaa7a7e67c73ea14f163c4d09f3835b94799d2458"}
Nov 02 01:42:09.440284 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:09.440236    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" podStartSLOduration=2.440221874 podStartE2EDuration="2.440221874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:09.388580221 +0000 UTC m=+1037.262221273" watchObservedRunningTime="2026-11-02 01:42:09.440221874 +0000 UTC m=+1037.313862925"
Nov 02 01:42:09.890811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:09.890780    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:09.892587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:09.892550    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 01:42:09.892742 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:09.892622    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 01:42:10.891454 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:10.891413    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 01:42:10.891882 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:10.891473    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 01:42:11.890991 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:11.890956    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 01:42:11.891164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:11.891006    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 01:42:12.891808 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:12.891771    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 01:42:12.892197 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:12.891834    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 01:42:13.891271 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:13.891240    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 01:42:13.891553 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:13.891307    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 01:42:15.006066 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:15.006030    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:15.007260 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:15.007227    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:15.007866 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:15.007843    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 01:42:16.038754 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:42:16.038715    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:42:21.987705 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:21.987675    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"]
Nov 02 01:42:21.991394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:21.991372    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.043878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.043839    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b71ba5-openshift-default-dockercfg-bzhxb\""
Nov 02 01:42:22.096638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.096580    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"]
Nov 02 01:42:22.106773 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.106739    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.106944 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.106774    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.106944 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.106849    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.107074 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.106939    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.107074 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.106987    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.107074 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.107033    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.107074 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.107066    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.107291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.107103    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.107291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.107122    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z58gz\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-kube-api-access-z58gz\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.207899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.207855    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.207899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.207900    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208150 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.207929    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208150 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.207952    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208150 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208128    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z58gz\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-kube-api-access-z58gz\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208253    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208290    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208454 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208350    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208454 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208380    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208454 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208394    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208677 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208647    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208703    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.208806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.208745    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.209020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.209001    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.210476 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.210453    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.210598 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.210581    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.272366 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.272327    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.273269 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.273237    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z58gz\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-kube-api-access-z58gz\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.282211 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.282183    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"]
Nov 02 01:42:22.287110 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.287086    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.304805 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.304782    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:22.342248 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.342215    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-187478-openshift-default-dockercfg-gsqtc\""
Nov 02 01:42:22.348041 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.348003    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"]
Nov 02 01:42:22.410381 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410345    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm66g\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-kube-api-access-gm66g\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.410565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410400    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.410565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410498    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.410565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410525    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb550fa5-5b81-4547-82c3-8fd91623df50-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.410709 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410668    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.410776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410759    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.410821 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410807    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.410940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410924    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.410990 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.410975    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.511950 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.511904    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.511950 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.511953    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512077    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm66g\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-kube-api-access-gm66g\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512129    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512165    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512181    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb550fa5-5b81-4547-82c3-8fd91623df50-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512222    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512298    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512348    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512358    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512568 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512522    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512696 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512675    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.512907 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.512885    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.513291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.513272    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb550fa5-5b81-4547-82c3-8fd91623df50-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.514688 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.514671    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.514917 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.514901    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.573106 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.573033    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.573248 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.573127    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm66g\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-kube-api-access-gm66g\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.602180 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.602153    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:22.737897 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.737872    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"]
Nov 02 01:42:22.739666 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:42:22.739640    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85f82d1_b482_4ae5_b9b8_e335e871f8e2.slice/crio-e5b093b84bae4ddd828b67c36080df05d9d4179c628cfd27ad8fda3c926deee1 WatchSource:0}: Error finding container e5b093b84bae4ddd828b67c36080df05d9d4179c628cfd27ad8fda3c926deee1: Status 404 returned error can't find the container with id e5b093b84bae4ddd828b67c36080df05d9d4179c628cfd27ad8fda3c926deee1
Nov 02 01:42:22.741945 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.741912    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:22.742025 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.741978    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:22.742025 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.742005    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:22.979806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.979782    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"]
Nov 02 01:42:22.981258 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:42:22.981226    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb550fa5_5b81_4547_82c3_8fd91623df50.slice/crio-11bfbf7f93bd6d799e8933c64952e604c12606baac11f12da2edd7496fe18507 WatchSource:0}: Error finding container 11bfbf7f93bd6d799e8933c64952e604c12606baac11f12da2edd7496fe18507: Status 404 returned error can't find the container with id 11bfbf7f93bd6d799e8933c64952e604c12606baac11f12da2edd7496fe18507
Nov 02 01:42:22.983763 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.983730    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:22.983876 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.983809    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:22.983876 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:22.983855    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:42:23.290523 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.290476    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" event={"ID":"fb550fa5-5b81-4547-82c3-8fd91623df50","Type":"ContainerStarted","Data":"6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec"}
Nov 02 01:42:23.290983 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.290550    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" event={"ID":"fb550fa5-5b81-4547-82c3-8fd91623df50","Type":"ContainerStarted","Data":"11bfbf7f93bd6d799e8933c64952e604c12606baac11f12da2edd7496fe18507"}
Nov 02 01:42:23.292525 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.292500    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" event={"ID":"a85f82d1-b482-4ae5-b9b8-e335e871f8e2","Type":"ContainerStarted","Data":"f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7"}
Nov 02 01:42:23.292642 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.292542    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" event={"ID":"a85f82d1-b482-4ae5-b9b8-e335e871f8e2","Type":"ContainerStarted","Data":"e5b093b84bae4ddd828b67c36080df05d9d4179c628cfd27ad8fda3c926deee1"}
Nov 02 01:42:23.305971 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.305944    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:23.307573 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.307550    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 01:42:23.307707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.307600    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 01:42:23.411840 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.411785    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" podStartSLOduration=1.411767931 podStartE2EDuration="1.411767931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:23.409364299 +0000 UTC m=+1051.283005364" watchObservedRunningTime="2026-11-02 01:42:23.411767931 +0000 UTC m=+1051.285408985"
Nov 02 01:42:23.537067 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.536634    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podStartSLOduration=2.53659062 podStartE2EDuration="2.53659062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:23.534075585 +0000 UTC m=+1051.407716638" watchObservedRunningTime="2026-11-02 01:42:23.53659062 +0000 UTC m=+1051.410231676"
Nov 02 01:42:23.603838 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:23.603729    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:24.306188 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:24.306138    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 01:42:24.306587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:24.306206    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 01:42:24.604264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:24.604170    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 01:42:24.604264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:24.604235    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 01:42:25.305779 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:25.305741    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 01:42:25.305957 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:25.305795    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 01:42:25.603274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:25.603177    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 01:42:25.603274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:25.603254    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 01:42:26.306415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:26.306118    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 01:42:26.306415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:26.306200    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 01:42:26.603242 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:26.603151    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 01:42:26.603242 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:26.603222    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 01:42:27.306166 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:27.306131    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 01:42:27.306345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:27.306198    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 01:42:27.603472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:27.603375    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 01:42:27.603472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:27.603450    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 01:42:28.306186 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:28.306145    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 01:42:28.306379 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:28.306200    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 01:42:28.603097 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:28.603008    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 02 01:42:28.603349 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:28.603321    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 02 01:42:29.306033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:29.305997    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 01:42:29.306572 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:29.306052    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 01:42:29.791350 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:29.791316    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:29.792258 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:29.792224    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:29.792996 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:29.792971    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 01:42:30.309688 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:30.309662    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:30.310153 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:30.310108    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:30.310709 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:42:30.310689    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 01:42:46.038066 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:42:46.038030    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:43:16.038489 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:43:16.038452    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:43:38.858258 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.858170    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"]
Nov 02 01:43:38.861757 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.861742    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.907744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.907714    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.907872 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.907749    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.907872 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.907777    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f9d676-48a6-4382-9ed7-0e9992dc925a-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.907996 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.907929    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.907996 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.907965    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.908098 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.908014    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.908154 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.908098    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.908154 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.908141    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4r2\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-kube-api-access-fm4r2\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.908274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.908256    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:38.913997 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.913974    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-88e76d-openshift-default-dockercfg-gsdsg\""
Nov 02 01:43:38.966267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:38.966243    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"]
Nov 02 01:43:39.008881 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.008856    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009036 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.008898    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4r2\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-kube-api-access-fm4r2\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009036 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.008957    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009036 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.008999    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009036 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.009027    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009273 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.009054    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f9d676-48a6-4382-9ed7-0e9992dc925a-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009273 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.009086    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009273 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.009113    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009273 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.009162    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.009724 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.009699    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.010131 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.010107    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.010212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.010125    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.010255 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.010226    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.010478 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.010452    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f9d676-48a6-4382-9ed7-0e9992dc925a-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.011812 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.011789    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.012237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.012218    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.069841 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.069812    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.069942 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.069888    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4r2\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-kube-api-access-fm4r2\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.174227 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.174160    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:39.261697 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.261669    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"]
Nov 02 01:43:39.265532 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.265511    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.312547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.312518    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.312713 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.312570    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.312713 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.312682    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.312816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.312728    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.312861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.312841    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2403ac43-089f-4aae-827e-5a59b8bf5a51-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.312909 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.312893    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.313024 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.313002    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx5jc\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-kube-api-access-lx5jc\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.313087 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.313061    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.313204 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.313184    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.329963 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.329935    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"]
Nov 02 01:43:39.414202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414172    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414355 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414219    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414355 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414271    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414355 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414291    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414355 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414306    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414355 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414331    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414355 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414349    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2403ac43-089f-4aae-827e-5a59b8bf5a51-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414381    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414425    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx5jc\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-kube-api-access-lx5jc\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414656    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414876 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414732    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.414876 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414840    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.415008 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.414986    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2403ac43-089f-4aae-827e-5a59b8bf5a51-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.415062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.415030    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.416797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.416778    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.417006 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.416990    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.475656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.475587    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.475769 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.475681    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx5jc\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-kube-api-access-lx5jc\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.579005 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.578974    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:39.600041 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:43:39.600014    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f9d676_48a6_4382_9ed7_0e9992dc925a.slice/crio-c329bd8e895acd6de56ee6ce3d595ffa4165234ffbf2b03ae99b3a0b02bd11c3 WatchSource:0}: Error finding container c329bd8e895acd6de56ee6ce3d595ffa4165234ffbf2b03ae99b3a0b02bd11c3: Status 404 returned error can't find the container with id c329bd8e895acd6de56ee6ce3d595ffa4165234ffbf2b03ae99b3a0b02bd11c3
Nov 02 01:43:39.600904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.600879    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"]
Nov 02 01:43:39.602500 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.602463    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:43:39.602564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.602545    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:43:39.602647 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.602591    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:43:39.709661 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.709625    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" event={"ID":"45f9d676-48a6-4382-9ed7-0e9992dc925a","Type":"ContainerStarted","Data":"0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702"}
Nov 02 01:43:39.709817 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.709677    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" event={"ID":"45f9d676-48a6-4382-9ed7-0e9992dc925a","Type":"ContainerStarted","Data":"c329bd8e895acd6de56ee6ce3d595ffa4165234ffbf2b03ae99b3a0b02bd11c3"}
Nov 02 01:43:39.830998 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.830941    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podStartSLOduration=1.83092669 podStartE2EDuration="1.83092669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:39.829096179 +0000 UTC m=+1127.702737231" watchObservedRunningTime="2026-11-02 01:43:39.83092669 +0000 UTC m=+1127.704567805"
Nov 02 01:43:39.962146 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.962105    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"]
Nov 02 01:43:39.963662 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:43:39.963636    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2403ac43_089f_4aae_827e_5a59b8bf5a51.slice/crio-5cb51cc62b2baa793109862d59a357aedc500f9de32dd81659f4fd52969caffe WatchSource:0}: Error finding container 5cb51cc62b2baa793109862d59a357aedc500f9de32dd81659f4fd52969caffe: Status 404 returned error can't find the container with id 5cb51cc62b2baa793109862d59a357aedc500f9de32dd81659f4fd52969caffe
Nov 02 01:43:39.966075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.966046    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:43:39.966171 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.966116    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:43:39.966171 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:39.966152    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:43:40.177873 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:40.177068    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:40.181663 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:40.181629    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 01:43:40.181797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:40.181687    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 01:43:40.716278 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:40.716240    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" event={"ID":"2403ac43-089f-4aae-827e-5a59b8bf5a51","Type":"ContainerStarted","Data":"d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c"}
Nov 02 01:43:40.716482 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:40.716300    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" event={"ID":"2403ac43-089f-4aae-827e-5a59b8bf5a51","Type":"ContainerStarted","Data":"5cb51cc62b2baa793109862d59a357aedc500f9de32dd81659f4fd52969caffe"}
Nov 02 01:43:40.843068 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:40.843014    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podStartSLOduration=1.842996532 podStartE2EDuration="1.842996532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:40.840626285 +0000 UTC m=+1128.714267349" watchObservedRunningTime="2026-11-02 01:43:40.842996532 +0000 UTC m=+1128.716637585"
Nov 02 01:43:41.176480 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:41.176023    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 01:43:41.176480 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:41.176101    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 01:43:41.585635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:41.584819    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:41.585635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:41.584979    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 01:43:41.585635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:41.585058    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 01:43:42.179636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:42.177097    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 01:43:42.179636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:42.177158    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 01:43:42.579635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:42.579549    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 01:43:42.579849 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:42.579649    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 01:43:43.178032 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:43.177992    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 01:43:43.178327 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:43.178282    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 01:43:43.580632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:43.579997    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 01:43:43.580632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:43.580070    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 01:43:44.176631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:44.175982    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 01:43:44.176631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:44.176061    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 01:43:44.583630 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:44.582967    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 01:43:44.583630 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:44.583031    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 01:43:45.174983 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:45.174948    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 01:43:45.175263 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:45.175236    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 01:43:45.596632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:45.596013    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 01:43:45.596632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:45.596075    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 01:43:46.039349 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:43:46.039309    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:43:46.175146 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:46.175113    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 02 01:43:46.175318 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:46.175180    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 02 01:43:46.579940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:46.579900    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 01:43:46.580122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:46.579980    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 01:43:47.346750 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:47.346716    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:47.347210 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:47.347145    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:47.349291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:47.349270    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 01:43:47.649698 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:47.649599    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:47.650048 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:47.650023    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:47.650836 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:47.650814    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:43:52.231393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.231356    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"]
Nov 02 01:43:52.236887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.236153    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.288463 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.288429    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tls-certs\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.288670 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.288504    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-home\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.288670 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.288595    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tmp-dir\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.288670 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.288647    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzvh2\" (UniqueName: \"kubernetes.io/projected/351cb013-4b1d-4d42-b1d8-9e45632c86d5-kube-api-access-mzvh2\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.288857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.288761    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-dshm\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.288857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.288792    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-model-cache\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.338581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.338550    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"openshift-service-ca.crt\""
Nov 02 01:43:52.338774 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.338710    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"distinct-w6-5888dc-kserve-self-signed-certs\""
Nov 02 01:43:52.338774 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.338710    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"default-dockercfg-mxnhx\""
Nov 02 01:43:52.338774 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.338739    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"kube-root-ca.crt\""
Nov 02 01:43:52.341358 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.341339    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"]
Nov 02 01:43:52.391190 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.391156    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-dshm\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.391514 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.391480    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-model-cache\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.391692 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.391674    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tls-certs\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.391797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.391750    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-home\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.391797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.391776    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tmp-dir\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.391920 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.391808    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzvh2\" (UniqueName: \"kubernetes.io/projected/351cb013-4b1d-4d42-b1d8-9e45632c86d5-kube-api-access-mzvh2\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.392503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.392478    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tmp-dir\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.392728 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.392573    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-home\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.394954 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.394931    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-model-cache\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.399191 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.399146    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tls-certs\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.406212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.406184    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-dshm\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.453855 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.453824    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzvh2\" (UniqueName: \"kubernetes.io/projected/351cb013-4b1d-4d42-b1d8-9e45632c86d5-kube-api-access-mzvh2\") pod \"distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.552685 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.552645    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:52.979556 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:52.979531    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"]
Nov 02 01:43:52.980912 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:43:52.980888    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod351cb013_4b1d_4d42_b1d8_9e45632c86d5.slice/crio-b3e30f2a2abe9c68a1279fd6a90f66d17445b1c6b582970ec6f7c1de8d3f5606 WatchSource:0}: Error finding container b3e30f2a2abe9c68a1279fd6a90f66d17445b1c6b582970ec6f7c1de8d3f5606: Status 404 returned error can't find the container with id b3e30f2a2abe9c68a1279fd6a90f66d17445b1c6b582970ec6f7c1de8d3f5606
Nov 02 01:43:53.862172 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:53.862135    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" event={"ID":"351cb013-4b1d-4d42-b1d8-9e45632c86d5","Type":"ContainerStarted","Data":"711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff"}
Nov 02 01:43:53.862172 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:53.862174    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" event={"ID":"351cb013-4b1d-4d42-b1d8-9e45632c86d5","Type":"ContainerStarted","Data":"b3e30f2a2abe9c68a1279fd6a90f66d17445b1c6b582970ec6f7c1de8d3f5606"}
Nov 02 01:43:53.986804 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:53.986754    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podStartSLOduration=1.986735533 podStartE2EDuration="1.986735533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:53.984706194 +0000 UTC m=+1141.858347246" watchObservedRunningTime="2026-11-02 01:43:53.986735533 +0000 UTC m=+1141.860376674"
Nov 02 01:43:54.553423 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:54.553392    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:43:57.203073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:57.203036    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"]
Nov 02 01:43:57.203497 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:57.203261    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy" containerID="cri-o://170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7" gracePeriod=30
Nov 02 01:43:59.958627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:59.958582    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"]
Nov 02 01:43:59.962458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:43:59.962442    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.066847 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.066818    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"openshift-service-ca.crt\""
Nov 02 01:44:00.067033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.066874    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"default-dockercfg-mqqzb\""
Nov 02 01:44:00.067033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.067012    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"distinct-w5-e06ea2-kserve-self-signed-certs\""
Nov 02 01:44:00.067150 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.067076    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"kube-root-ca.crt\""
Nov 02 01:44:00.069662 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.069638    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"]
Nov 02 01:44:00.072525 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.072509    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-model-cache\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.072592 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.072548    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tls-certs\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.072592 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.072576    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zxz\" (UniqueName: \"kubernetes.io/projected/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-kube-api-access-w2zxz\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.072732 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.072713    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-home\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.072807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.072784    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-dshm\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.072910 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.072894    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tmp-dir\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.173586 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.173549    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tmp-dir\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.173792 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.173684    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-model-cache\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.173846 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.173790    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tls-certs\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.173846 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.173828    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zxz\" (UniqueName: \"kubernetes.io/projected/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-kube-api-access-w2zxz\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.173927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.173868    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-home\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.173927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.173907    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-dshm\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.174047 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.174026    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tmp-dir\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.174104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.174044    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-model-cache\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.174253 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.174234    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-home\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.176220 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.176190    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-dshm\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.176554 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.176538    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tls-certs\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.240737 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.234724    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zxz\" (UniqueName: \"kubernetes.io/projected/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-kube-api-access-w2zxz\") pod \"distinct-w5-e06ea2-kserve-76966cdcb7-kflcq\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") " pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.275030 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.274987    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:00.688198 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.688159    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"]
Nov 02 01:44:00.689487 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:44:00.689457    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a1a976f_fa19_4dad_a28c_5a8039ff41ae.slice/crio-bfb0ce9c04490394ea4d266d28fa558eab74eac71cadcaf406d1e5b8f0f04445 WatchSource:0}: Error finding container bfb0ce9c04490394ea4d266d28fa558eab74eac71cadcaf406d1e5b8f0f04445: Status 404 returned error can't find the container with id bfb0ce9c04490394ea4d266d28fa558eab74eac71cadcaf406d1e5b8f0f04445
Nov 02 01:44:00.902282 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.902240    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" event={"ID":"6a1a976f-fa19-4dad-a28c-5a8039ff41ae","Type":"ContainerStarted","Data":"550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4"}
Nov 02 01:44:00.902453 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:00.902292    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" event={"ID":"6a1a976f-fa19-4dad-a28c-5a8039ff41ae","Type":"ContainerStarted","Data":"bfb0ce9c04490394ea4d266d28fa558eab74eac71cadcaf406d1e5b8f0f04445"}
Nov 02 01:44:01.047451 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:01.047395    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podStartSLOduration=2.047381242 podStartE2EDuration="2.047381242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:01.044526678 +0000 UTC m=+1148.918167727" watchObservedRunningTime="2026-11-02 01:44:01.047381242 +0000 UTC m=+1148.921022293"
Nov 02 01:44:01.275574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:01.275540    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:02.517457 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.517410    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:44:02.556191 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.555309    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:44:02.576159 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.576082    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:44:02.622956 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.622926    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-envoy\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.623203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.623183    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77a81fde-83aa-47ba-95da-64721bc396e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a81fde-83aa-47ba-95da-64721bc396e0-istio-podinfo\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.623319 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.623307    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-data\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.623426 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.623413    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a81fde-83aa-47ba-95da-64721bc396e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a81fde-83aa-47ba-95da-64721bc396e0-istiod-ca-cert\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.623593 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.623579    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-kube-api-access-dvqfl\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-kube-api-access-dvqfl\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.623728 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.623715    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-istio-token\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-istio-token\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.623834 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.623823    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-certs\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.623940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.623930    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-socket\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.625151 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.624381    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-data" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:02.625151 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.624551    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-certs" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:02.625329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.624566    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-socket" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:02.625329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.624796    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a81fde-83aa-47ba-95da-64721bc396e0-istiod-ca-cert" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:44:02.625329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.625217    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-credential-socket\") pod \"77a81fde-83aa-47ba-95da-64721bc396e0\" (UID: \"77a81fde-83aa-47ba-95da-64721bc396e0\") "
Nov 02 01:44:02.625649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.625474    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-credential-socket" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:02.626982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.626955    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.627086 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.626986    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.627086 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.627000    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.627086 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.627014    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.627086 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.627026    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a81fde-83aa-47ba-95da-64721bc396e0-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.627287 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.627266    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-envoy" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:44:02.627500 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.627486    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77a81fde-83aa-47ba-95da-64721bc396e0-istio-podinfo" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:44:02.627598 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.627571    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-istio-token" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:02.627675 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.627663    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-kube-api-access-dvqfl" pod "77a81fde-83aa-47ba-95da-64721bc396e0" (UID: "77a81fde-83aa-47ba-95da-64721bc396e0"). InnerVolumeSpecName "kube-api-access-dvqfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:44:02.728501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.728464    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqfl\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-kube-api-access-dvqfl\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.728501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.728493    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a81fde-83aa-47ba-95da-64721bc396e0-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.728501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.728503    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a81fde-83aa-47ba-95da-64721bc396e0-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.728501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.728511    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a81fde-83aa-47ba-95da-64721bc396e0-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:44:02.919209 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.919173    4287 generic.go:356] "Generic (PLEG): container finished" podID="77a81fde-83aa-47ba-95da-64721bc396e0" containerID="170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7" exitCode=0
Nov 02 01:44:02.919394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.919229    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"
Nov 02 01:44:02.919394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.919260    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" event={"ID":"77a81fde-83aa-47ba-95da-64721bc396e0","Type":"ContainerDied","Data":"170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7"}
Nov 02 01:44:02.919394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.919324    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj" event={"ID":"77a81fde-83aa-47ba-95da-64721bc396e0","Type":"ContainerDied","Data":"e56d1d37e3ff4b65b1c39ff2df096b16b246761bbb181ff496967b28eab162ff"}
Nov 02 01:44:02.919394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.919349    4287 scope.go:122] "RemoveContainer" containerID="170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7"
Nov 02 01:44:02.930793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.930775    4287 scope.go:122] "RemoveContainer" containerID="170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7"
Nov 02 01:44:02.931083 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:44:02.931066    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7\": container with ID starting with 170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7 not found: ID does not exist" containerID="170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7"
Nov 02 01:44:02.931144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.931093    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7"} err="failed to get container status \"170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7\": rpc error: code = NotFound desc = could not find container \"170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7\": container with ID starting with 170d6aa7000786fcd3a254e23c8a013f04f5573af633aec73b548d4987c364d7 not found: ID does not exist"
Nov 02 01:44:02.933033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:02.933016    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:44:03.203779 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:03.203706    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"]
Nov 02 01:44:03.209159 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:03.209128    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-b8wvj"]
Nov 02 01:44:04.238601 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:04.238568    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" path="/var/lib/kubelet/pods/77a81fde-83aa-47ba-95da-64721bc396e0/volumes"
Nov 02 01:44:10.275326 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:10.275289    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:10.287388 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:10.287366    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:10.973342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:10.973314    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:44:13.435475 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.435433    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"]
Nov 02 01:44:13.435914 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.435900    4287 cpu_manager.go:414] "containerMap: removing container" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy"
Nov 02 01:44:13.435957 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.435916    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy"
Nov 02 01:44:13.436017 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.436007    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a81fde-83aa-47ba-95da-64721bc396e0" containerName="istio-proxy"
Nov 02 01:44:13.438880 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.438862    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.532529 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.532493    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-home\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.532699 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.532650    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxg4m\" (UniqueName: \"kubernetes.io/projected/05a9650a-d6e0-4509-a65b-4b60e98107da-kube-api-access-gxg4m\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.532699 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.532697    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-tmp-dir\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.532840 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.532717    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a9650a-d6e0-4509-a65b-4b60e98107da-tls-certs\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.532840 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.532809    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-dshm\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.532917 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.532881    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-model-cache\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.540279 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.540261    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"default-dockercfg-2rrmt\""
Nov 02 01:44:13.540356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.540343    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"openshift-service-ca.crt\""
Nov 02 01:44:13.540394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.540368    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"distinct-w3-88e76d-kserve-self-signed-certs\""
Nov 02 01:44:13.540431 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.540387    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"kube-root-ca.crt\""
Nov 02 01:44:13.543168 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.543152    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"]
Nov 02 01:44:13.633406 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.633370    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxg4m\" (UniqueName: \"kubernetes.io/projected/05a9650a-d6e0-4509-a65b-4b60e98107da-kube-api-access-gxg4m\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.633594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.633419    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-tmp-dir\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.633594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.633450    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a9650a-d6e0-4509-a65b-4b60e98107da-tls-certs\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.633594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.633486    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-dshm\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.633594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.633571    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-model-cache\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.633594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.633591    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-home\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.633969 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.633947    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-tmp-dir\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.634031 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.633977    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-model-cache\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.634963 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.634940    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-home\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.636062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.636041    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-dshm\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.636317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.636300    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a9650a-d6e0-4509-a65b-4b60e98107da-tls-certs\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.694092 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.694032    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxg4m\" (UniqueName: \"kubernetes.io/projected/05a9650a-d6e0-4509-a65b-4b60e98107da-kube-api-access-gxg4m\") pod \"distinct-w3-88e76d-kserve-65fc8fcbbd-m989s\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:13.750033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:13.749988    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:14.165764 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:14.165740    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"]
Nov 02 01:44:14.166754 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:44:14.166726    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05a9650a_d6e0_4509_a65b_4b60e98107da.slice/crio-6d1c385bc20e4d949eadeae8f5e0b861b627d068b373a2e72ab2a2bed8f6c32d WatchSource:0}: Error finding container 6d1c385bc20e4d949eadeae8f5e0b861b627d068b373a2e72ab2a2bed8f6c32d: Status 404 returned error can't find the container with id 6d1c385bc20e4d949eadeae8f5e0b861b627d068b373a2e72ab2a2bed8f6c32d
Nov 02 01:44:14.993267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:14.993225    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" event={"ID":"05a9650a-d6e0-4509-a65b-4b60e98107da","Type":"ContainerStarted","Data":"e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a"}
Nov 02 01:44:14.993267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:14.993266    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" event={"ID":"05a9650a-d6e0-4509-a65b-4b60e98107da","Type":"ContainerStarted","Data":"6d1c385bc20e4d949eadeae8f5e0b861b627d068b373a2e72ab2a2bed8f6c32d"}
Nov 02 01:44:15.114584 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:15.114529    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podStartSLOduration=2.114516111 podStartE2EDuration="2.114516111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:15.112130364 +0000 UTC m=+1162.985771416" watchObservedRunningTime="2026-11-02 01:44:15.114516111 +0000 UTC m=+1162.988157163"
Nov 02 01:44:15.751116 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:15.751074    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:16.038293 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:44:16.038255    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:44:23.750694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:23.750660    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:23.772649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:23.766054    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:23.949382 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:23.949346    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"]
Nov 02 01:44:23.953060 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:23.953040    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.015482 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.015452    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"]
Nov 02 01:44:24.040712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.040671    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.040862 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.040742    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.040921 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.040898    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.040958 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.040940    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.041006 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.040968    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.041071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.041047    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45fn\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-kube-api-access-w45fn\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.041121 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.041104    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301a31fd-361d-4f0b-b3c6-e46374bd5094-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.041223 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.041195    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.041346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.041309    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.056011 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.055976    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:44:24.142066 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142030    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142081    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142121    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142210    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142232    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142466 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142256    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142466 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142282    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w45fn\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-kube-api-access-w45fn\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142580 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142529    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142580 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142561    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301a31fd-361d-4f0b-b3c6-e46374bd5094-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142716 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142629    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142716 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142602    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142716 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142600    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.142975 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.142951    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.143214 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.143159    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301a31fd-361d-4f0b-b3c6-e46374bd5094-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.144752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.144731    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.145126 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.145103    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.204895 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.204864    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.205477 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.205449    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45fn\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-kube-api-access-w45fn\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.266993 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.266920    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:24.696567 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.696544    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"]
Nov 02 01:44:24.698142 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:44:24.698116    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301a31fd_361d_4f0b_b3c6_e46374bd5094.slice/crio-b6781e911d544a0183a48292831148986fabba864797b32364c96a251dbf13cd WatchSource:0}: Error finding container b6781e911d544a0183a48292831148986fabba864797b32364c96a251dbf13cd: Status 404 returned error can't find the container with id b6781e911d544a0183a48292831148986fabba864797b32364c96a251dbf13cd
Nov 02 01:44:24.700394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.700361    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:24.700483 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.700423    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:24.700483 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:24.700458    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:44:25.053929 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:25.053887    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" event={"ID":"301a31fd-361d-4f0b-b3c6-e46374bd5094","Type":"ContainerStarted","Data":"7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab"}
Nov 02 01:44:25.054379 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:25.053981    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" event={"ID":"301a31fd-361d-4f0b-b3c6-e46374bd5094","Type":"ContainerStarted","Data":"b6781e911d544a0183a48292831148986fabba864797b32364c96a251dbf13cd"}
Nov 02 01:44:25.175600 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:25.175546    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podStartSLOduration=2.17553112 podStartE2EDuration="2.17553112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:25.173320144 +0000 UTC m=+1173.046961197" watchObservedRunningTime="2026-11-02 01:44:25.17553112 +0000 UTC m=+1173.049172233"
Nov 02 01:44:25.267978 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:25.267946    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:25.269348 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:25.269326    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 01:44:25.269478 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:25.269369    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 01:44:26.268632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:26.268010    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 01:44:26.268632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:26.268087    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 01:44:27.270055 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:27.270003    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 01:44:27.270723 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:27.270689    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 01:44:28.271632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:28.270979    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 01:44:28.271632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:28.271079    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 01:44:29.277632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:29.272187    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 01:44:29.277632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:29.272258    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 01:44:30.274649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:30.274001    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 01:44:30.274649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:30.274067    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 01:44:31.267405 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:31.267370    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 01:44:31.267840 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:31.267426    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 01:44:32.272027 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:32.271990    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:32.272597 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:32.272546    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:32.273120 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:32.273100    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:44:38.038442 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.038400    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"]
Nov 02 01:44:38.043424 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.043405    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.101853 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.101823    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"]
Nov 02 01:44:38.116855 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.116822    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-home\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.117005 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.116871    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tmp-dir\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.117005 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.116975    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-dshm\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.117130 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.117004    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-model-cache\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.117187 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.117130    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbvj\" (UniqueName: \"kubernetes.io/projected/9b097b84-fb8c-41f6-8b74-9f9a29245b59-kube-api-access-wxbvj\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.117187 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.117170    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tls-certs\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.147074 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.147048    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"distinct-w2-36a1e0-kserve-self-signed-certs\""
Nov 02 01:44:38.217807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.217769    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbvj\" (UniqueName: \"kubernetes.io/projected/9b097b84-fb8c-41f6-8b74-9f9a29245b59-kube-api-access-wxbvj\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.217962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.217828    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tls-certs\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.217962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.217894    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-home\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.217962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.217917    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tmp-dir\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.217962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.217940    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-dshm\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.217962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.217956    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-model-cache\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.218353 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.218334    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-model-cache\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.218452 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.218432    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-home\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.218514 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.218484    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tmp-dir\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.220288 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.220270    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-dshm\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.220395 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.220372    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tls-certs\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.281083 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.281050    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbvj\" (UniqueName: \"kubernetes.io/projected/9b097b84-fb8c-41f6-8b74-9f9a29245b59-kube-api-access-wxbvj\") pod \"distinct-w2-36a1e0-kserve-84b85f9f77-jmg42\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.358213 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.358114    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:38.741038 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:38.741014    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"]
Nov 02 01:44:38.742646 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:44:38.742597    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b097b84_fb8c_41f6_8b74_9f9a29245b59.slice/crio-da818661fcc9129e750b6fa1e36b69bfac51c8c5b158377460217ac455434462 WatchSource:0}: Error finding container da818661fcc9129e750b6fa1e36b69bfac51c8c5b158377460217ac455434462: Status 404 returned error can't find the container with id da818661fcc9129e750b6fa1e36b69bfac51c8c5b158377460217ac455434462
Nov 02 01:44:39.158306 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:39.158264    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" event={"ID":"9b097b84-fb8c-41f6-8b74-9f9a29245b59","Type":"ContainerStarted","Data":"b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709"}
Nov 02 01:44:39.158736 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:39.158324    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" event={"ID":"9b097b84-fb8c-41f6-8b74-9f9a29245b59","Type":"ContainerStarted","Data":"da818661fcc9129e750b6fa1e36b69bfac51c8c5b158377460217ac455434462"}
Nov 02 01:44:39.358660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:39.358630    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:46.038531 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:44:46.038497    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:44:48.359219 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:48.359174    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:48.371976 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:48.371954    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:48.489773 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:48.489723    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podStartSLOduration=11.4897104 podStartE2EDuration="11.4897104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:39.274913952 +0000 UTC m=+1187.148555006" watchObservedRunningTime="2026-11-02 01:44:48.4897104 +0000 UTC m=+1196.363351452"
Nov 02 01:44:49.233854 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:49.233826    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:44:52.356022 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:52.355992    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:44:52.360389 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:44:52.360368    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:45:08.445793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.445755    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"]
Nov 02 01:45:08.451185 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.451158    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.551749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.551716    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-tmp-dir\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.551749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.551747    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrfmq\" (UniqueName: \"kubernetes.io/projected/94488447-7e8f-4b85-b491-d3b28bf680ed-kube-api-access-rrfmq\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.551942 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.551848    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-home\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.551942 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.551926    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-model-cache\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.552007 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.551994    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-dshm\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.552120 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.552101    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94488447-7e8f-4b85-b491-d3b28bf680ed-tls-certs\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.553582 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.553557    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"kube-root-ca.crt\""
Nov 02 01:45:08.554151 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.554117    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"openshift-service-ca.crt\""
Nov 02 01:45:08.554223 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.554172    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"default-dockercfg-sphkv\""
Nov 02 01:45:08.554293 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.554276    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"distinct-2-w1-f276c3-kserve-self-signed-certs\""
Nov 02 01:45:08.560743 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.560711    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"]
Nov 02 01:45:08.652793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.652759    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94488447-7e8f-4b85-b491-d3b28bf680ed-tls-certs\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.653001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.652821    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-tmp-dir\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.653001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.652879    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrfmq\" (UniqueName: \"kubernetes.io/projected/94488447-7e8f-4b85-b491-d3b28bf680ed-kube-api-access-rrfmq\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.653001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.652938    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-home\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.653166 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.653008    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-model-cache\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.653166 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.653054    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-dshm\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.653333 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.653303    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-tmp-dir\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.653477 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.653346    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-home\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.653477 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.653432    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-model-cache\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.655329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.655309    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-dshm\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.655453 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.655376    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94488447-7e8f-4b85-b491-d3b28bf680ed-tls-certs\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.716295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.716213    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrfmq\" (UniqueName: \"kubernetes.io/projected/94488447-7e8f-4b85-b491-d3b28bf680ed-kube-api-access-rrfmq\") pod \"distinct-2-w1-f276c3-kserve-74c765996d-rl9dh\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") " pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:08.763089 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:08.763054    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:09.187995 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:09.187962    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"]
Nov 02 01:45:09.189418 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:45:09.189391    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94488447_7e8f_4b85_b491_d3b28bf680ed.slice/crio-1be66737e60edcaeaefd9280bab921bbed2e901bff82168ed5a5e33e95d924e0 WatchSource:0}: Error finding container 1be66737e60edcaeaefd9280bab921bbed2e901bff82168ed5a5e33e95d924e0: Status 404 returned error can't find the container with id 1be66737e60edcaeaefd9280bab921bbed2e901bff82168ed5a5e33e95d924e0
Nov 02 01:45:09.355073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:09.354936    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" event={"ID":"94488447-7e8f-4b85-b491-d3b28bf680ed","Type":"ContainerStarted","Data":"25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41"}
Nov 02 01:45:09.355073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:09.354987    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" event={"ID":"94488447-7e8f-4b85-b491-d3b28bf680ed","Type":"ContainerStarted","Data":"1be66737e60edcaeaefd9280bab921bbed2e901bff82168ed5a5e33e95d924e0"}
Nov 02 01:45:09.474000 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:09.473891    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podStartSLOduration=1.473876569 podStartE2EDuration="1.473876569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:09.471123021 +0000 UTC m=+1217.344764074" watchObservedRunningTime="2026-11-02 01:45:09.473876569 +0000 UTC m=+1217.347517620"
Nov 02 01:45:09.763550 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:09.763520    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:16.040095 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:45:16.040050    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:45:18.764319 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:18.764275    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:18.777167 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:18.777139    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:19.427628 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:19.427574    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:45:38.225660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.224116    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"]
Nov 02 01:45:38.230048 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.230023    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.275071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.275036    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"]
Nov 02 01:45:38.314097 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314041    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.314307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314125    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb8tx\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-kube-api-access-wb8tx\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.314307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314171    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.314307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314263    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.314307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314296    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.314565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314313    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.314565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314372    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3435e444-4c0f-441b-a1c3-8623ce6b05db-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.314565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314407    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.314565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.314453    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.318493 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.318466    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f276c3-openshift-default-dockercfg-72fq4\""
Nov 02 01:45:38.416023 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.415981    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416037    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416065    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416111    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3435e444-4c0f-441b-a1c3-8623ce6b05db-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416141    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416416    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416495 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416478    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416553 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416489    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416555    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8tx\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-kube-api-access-wb8tx\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416692 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416650    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416833 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416799    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416912 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416880    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.416973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416949    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.417028 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.416994    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3435e444-4c0f-441b-a1c3-8623ce6b05db-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.418712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.418678    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.419160 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.419141    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.478544 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.478452    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.478544 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.478487    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8tx\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-kube-api-access-wb8tx\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.545895 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.545860    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:38.991744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.991714    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"]
Nov 02 01:45:38.993011 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:45:38.992983    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3435e444_4c0f_441b_a1c3_8623ce6b05db.slice/crio-f74207ae52a8373282913582e6980cdd9ff4fd934f7c125d42f0273a75aa79dc WatchSource:0}: Error finding container f74207ae52a8373282913582e6980cdd9ff4fd934f7c125d42f0273a75aa79dc: Status 404 returned error can't find the container with id f74207ae52a8373282913582e6980cdd9ff4fd934f7c125d42f0273a75aa79dc
Nov 02 01:45:38.995641 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.995572    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:38.995718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.995679    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:38.995718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:38.995709    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:45:39.578557 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:39.578522    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" event={"ID":"3435e444-4c0f-441b-a1c3-8623ce6b05db","Type":"ContainerStarted","Data":"25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609"}
Nov 02 01:45:39.579042 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:39.578572    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" event={"ID":"3435e444-4c0f-441b-a1c3-8623ce6b05db","Type":"ContainerStarted","Data":"f74207ae52a8373282913582e6980cdd9ff4fd934f7c125d42f0273a75aa79dc"}
Nov 02 01:45:39.702395 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:39.702342    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podStartSLOduration=1.702315558 podStartE2EDuration="1.702315558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:39.698427908 +0000 UTC m=+1247.572068979" watchObservedRunningTime="2026-11-02 01:45:39.702315558 +0000 UTC m=+1247.575956610"
Nov 02 01:45:40.546872 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:40.546832    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:40.548482 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:40.548455    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 01:45:40.548644 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:40.548506    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 01:45:41.547638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:41.547576    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 01:45:41.548129 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:41.547668    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 01:45:42.546768 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:42.546735    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 01:45:42.546944 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:42.546790    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 01:45:43.547705 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:43.547671    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 01:45:43.548102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:43.547735    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 01:45:44.547069 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:44.547034    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 01:45:44.547248 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:44.547099    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 01:45:45.547221 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:45.547190    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 01:45:45.547596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:45.547271    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 01:45:46.038935 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:45:46.038901    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:45:46.583311 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:46.583279    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:46.584129 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:46.584103    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:45:46.585301 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:45:46.585269    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:46:16.040253 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:46:16.040214    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:46:37.711932 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.711854    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"]
Nov 02 01:46:37.715473 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.715457    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.770527 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.770477    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"openshift-service-ca.crt\""
Nov 02 01:46:37.801459 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.801431    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-model-cache\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.801600 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.801463    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-home\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.801600 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.801573    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950028ae-303e-476d-9271-052f2bd8db10-tls-certs\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.801600 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.801601    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25krh\" (UniqueName: \"kubernetes.io/projected/950028ae-303e-476d-9271-052f2bd8db10-kube-api-access-25krh\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.801734 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.801713    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-tmp-dir\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.801772 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.801754    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-dshm\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.819881 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.819858    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"kube-root-ca.crt\""
Nov 02 01:46:37.820031 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.820016    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"default-dockercfg-qnkcp\""
Nov 02 01:46:37.820111 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.820087    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"distinct-2-w4-187478-kserve-self-signed-certs\""
Nov 02 01:46:37.823145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.823124    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"]
Nov 02 01:46:37.903314 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.903286    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-dshm\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.903496 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.903370    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-model-cache\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.903496 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.903392    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-home\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.903496 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.903433    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950028ae-303e-476d-9271-052f2bd8db10-tls-certs\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.903711 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.903632    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25krh\" (UniqueName: \"kubernetes.io/projected/950028ae-303e-476d-9271-052f2bd8db10-kube-api-access-25krh\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.903787 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.903737    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-tmp-dir\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.903846 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.903820    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-model-cache\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.903902 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.903848    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-home\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.904086 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.904050    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-tmp-dir\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.906464 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.906435    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-dshm\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.909300 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.909277    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950028ae-303e-476d-9271-052f2bd8db10-tls-certs\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:37.965304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:37.965237    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25krh\" (UniqueName: \"kubernetes.io/projected/950028ae-303e-476d-9271-052f2bd8db10-kube-api-access-25krh\") pod \"distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") " pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:38.027568 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:38.027532    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:38.458940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:38.458912    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"]
Nov 02 01:46:38.460260 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:46:38.460229    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod950028ae_303e_476d_9271_052f2bd8db10.slice/crio-5a484736a05787f3d15288e34e53dadedb6c7358edef534a139592625ac96b09 WatchSource:0}: Error finding container 5a484736a05787f3d15288e34e53dadedb6c7358edef534a139592625ac96b09: Status 404 returned error can't find the container with id 5a484736a05787f3d15288e34e53dadedb6c7358edef534a139592625ac96b09
Nov 02 01:46:38.938675 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:38.938631    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" event={"ID":"950028ae-303e-476d-9271-052f2bd8db10","Type":"ContainerStarted","Data":"7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d"}
Nov 02 01:46:38.938675 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:38.938672    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" event={"ID":"950028ae-303e-476d-9271-052f2bd8db10","Type":"ContainerStarted","Data":"5a484736a05787f3d15288e34e53dadedb6c7358edef534a139592625ac96b09"}
Nov 02 01:46:39.028697 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:39.028668    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:39.058885 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:39.058836    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podStartSLOduration=2.058821933 podStartE2EDuration="2.058821933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:39.056393991 +0000 UTC m=+1306.930035044" watchObservedRunningTime="2026-11-02 01:46:39.058821933 +0000 UTC m=+1306.932462985"
Nov 02 01:46:46.046035 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:46:46.045993    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:46:48.028466 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:48.028433    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:48.041446 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:48.041421    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:46:49.019999 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:46:49.019965    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:47:16.168246 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:47:16.168204    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:47:16.357467 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:16.357431    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:16.357813 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:16.357781    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:24.325276 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.325243    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"]
Nov 02 01:47:24.328930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.328914    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.408701 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.408674    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.408836 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.408781    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.408836 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.408810    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644676-76e5-48ca-8542-9dc6af47e394-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.408935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.408915    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8rp\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-kube-api-access-rp8rp\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.408976 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.408945    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.409009 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.408978    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.409042 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.409023    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.409075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.409046    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.409170 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.409151    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644676-76e5-48ca-8542-9dc6af47e394-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.425049 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.425022    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"]
Nov 02 01:47:24.510420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510394    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8rp\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-kube-api-access-rp8rp\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.510594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510425    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.510594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510448    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.510594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510478    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.510594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510493    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.510594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510533    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644676-76e5-48ca-8542-9dc6af47e394-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.510929 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510621    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.510929 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510693    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.510929 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510710    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644676-76e5-48ca-8542-9dc6af47e394-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.511139 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.510959    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.511139 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.511036    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.511139 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.511123    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.511306 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.511158    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.511411 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.511390    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644676-76e5-48ca-8542-9dc6af47e394-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.512851 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.512831    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.513123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.513094    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644676-76e5-48ca-8542-9dc6af47e394-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.571322 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.571299    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.571481 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.571462    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8rp\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-kube-api-access-rp8rp\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:24.641638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:24.641552    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:25.075058 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:25.075029    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"]
Nov 02 01:47:25.075182 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:47:25.075098    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67644676_76e5_48ca_8542_9dc6af47e394.slice/crio-2777acfb1ce61b4419edde8a637ba0ceeab2ff20f6e992c143db09e88aa8b731 WatchSource:0}: Error finding container 2777acfb1ce61b4419edde8a637ba0ceeab2ff20f6e992c143db09e88aa8b731: Status 404 returned error can't find the container with id 2777acfb1ce61b4419edde8a637ba0ceeab2ff20f6e992c143db09e88aa8b731
Nov 02 01:47:25.077438 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:25.077409    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:25.077539 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:25.077473    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:25.077539 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:25.077504    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:25.372447 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:25.372358    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" event={"ID":"67644676-76e5-48ca-8542-9dc6af47e394","Type":"ContainerStarted","Data":"6a2bbe5c98ea9103d7f78c01bf5b4f2498b22aa8294a174696825a1f386508a2"}
Nov 02 01:47:25.372447 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:25.372428    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" event={"ID":"67644676-76e5-48ca-8542-9dc6af47e394","Type":"ContainerStarted","Data":"2777acfb1ce61b4419edde8a637ba0ceeab2ff20f6e992c143db09e88aa8b731"}
Nov 02 01:47:25.496400 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:25.496344    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podStartSLOduration=1.496325567 podStartE2EDuration="1.496325567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:25.493307226 +0000 UTC m=+1353.366948279" watchObservedRunningTime="2026-11-02 01:47:25.496325567 +0000 UTC m=+1353.369966618"
Nov 02 01:47:25.642503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:25.642415    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:26.643621 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:26.643573    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:47:26.643996 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:26.643637    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 01:47:27.641999 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:27.641963    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:47:27.642177 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:27.642021    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 01:47:28.642908 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:28.642870    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:47:28.643377 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:28.642931    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 01:47:29.641938 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:29.641904    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:47:29.642118 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:29.641950    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 01:47:30.642894 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:30.642841    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:47:30.643318 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:30.642917    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 01:47:31.642699 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:31.642667    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 01:47:31.642847 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:31.642721    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 01:47:32.646217 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:32.646187    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:32.646656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:32.646571    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:32.647274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:32.647257    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:47:39.169448 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.169411    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"]
Nov 02 01:47:39.173481 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.173463    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.267797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.267763    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"]
Nov 02 01:47:39.305288 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305257    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4cbb\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-kube-api-access-r4cbb\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.305443 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305365    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.305443 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305407    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.305443 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305430    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.305543 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305516    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.305655 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305635    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.305690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305659    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.305787 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305767    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.305822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.305805    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.406784 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.406756    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.406931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.406793    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.406931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.406850    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.406931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.406876    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.406931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.406912    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4cbb\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-kube-api-access-r4cbb\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407116 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.406973    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407116 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.407020    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.407186    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.407196    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407321 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.407263    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407321 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.407280    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407426 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.407385    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407500 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.407480    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.407812 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.407795    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.409338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.409314    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.409588 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.409570    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.467887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.467831    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4cbb\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-kube-api-access-r4cbb\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.467887 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.467870    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.486828 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.486810    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:39.919273 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:47:39.919239    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1e58e2_df90_44dd_bf75_656c94f4bcdc.slice/crio-a03b8d43d35afa1e11ad52991b3c674419ae9f4e53189bbc36f9e8946a04af89 WatchSource:0}: Error finding container a03b8d43d35afa1e11ad52991b3c674419ae9f4e53189bbc36f9e8946a04af89: Status 404 returned error can't find the container with id a03b8d43d35afa1e11ad52991b3c674419ae9f4e53189bbc36f9e8946a04af89
Nov 02 01:47:39.919864 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.919768    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"]
Nov 02 01:47:39.921907 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.921867    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:39.921993 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.921949    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:39.922033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:39.921987    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:40.179521 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.179443    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"]
Nov 02 01:47:40.183429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.183409    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.243894 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.243859    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"]
Nov 02 01:47:40.318375 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.318337    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.318574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.318414    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.318574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.318446    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhtbd\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-kube-api-access-fhtbd\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.318574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.318483    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.318574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.318517    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.318822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.318632    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.318822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.318698    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.318822 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.318745    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.319094 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.319071    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420039 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420007    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420055    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420116    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420185    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420359 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420310    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420375    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhtbd\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-kube-api-access-fhtbd\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420480 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420444    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420538 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420497    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420540    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420704 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420635    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420704 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420649    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.420805 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420786    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.421005 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.420982    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.421266 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.421250    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.422923 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.422903    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.423066 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.423049    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.468393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.468318    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" event={"ID":"dc1e58e2-df90-44dd-bf75-656c94f4bcdc","Type":"ContainerStarted","Data":"cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6"}
Nov 02 01:47:40.468393 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.468372    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" event={"ID":"dc1e58e2-df90-44dd-bf75-656c94f4bcdc","Type":"ContainerStarted","Data":"a03b8d43d35afa1e11ad52991b3c674419ae9f4e53189bbc36f9e8946a04af89"}
Nov 02 01:47:40.481033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.481006    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.481033 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.481026    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhtbd\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-kube-api-access-fhtbd\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.487927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.487907    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:40.489817 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.489792    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:47:40.489922 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.489844    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 01:47:40.497013 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.496992    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:40.590490 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.590425    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podStartSLOduration=1.5904067290000001 podStartE2EDuration="1.590406729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:40.586919282 +0000 UTC m=+1368.460560336" watchObservedRunningTime="2026-11-02 01:47:40.590406729 +0000 UTC m=+1368.464047781"
Nov 02 01:47:40.935741 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.935717    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"]
Nov 02 01:47:40.937640 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:47:40.937596    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8276d1e_3e79_4fa2_8688_14f3da5c3a92.slice/crio-7497cfedf2565eb8f5d00ea6d50e46ee734285837c7b6686f00ebea83ac1455c WatchSource:0}: Error finding container 7497cfedf2565eb8f5d00ea6d50e46ee734285837c7b6686f00ebea83ac1455c: Status 404 returned error can't find the container with id 7497cfedf2565eb8f5d00ea6d50e46ee734285837c7b6686f00ebea83ac1455c
Nov 02 01:47:40.940236 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.940198    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:40.940310 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.940263    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:40.940310 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:40.940301    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:41.108149 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.108118    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"]
Nov 02 01:47:41.111917 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.111897    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.203781 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.203690    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"]
Nov 02 01:47:41.228814 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.228776    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.228954 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.228876    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.229005 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.228979    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhn5r\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-kube-api-access-zhn5r\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.229059 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.229035    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.229102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.229093    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.229153 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.229121    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.229251 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.229231    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.229359 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.229286    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.229359 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.229318    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.330676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.330640    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.330676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.330680    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.330904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.330735    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.330904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.330768    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.330904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.330802    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.330904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.330867    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.330904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.330892    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.331181 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.330976    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn5r\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-kube-api-access-zhn5r\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.331181 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.331035    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.331291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.331185    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.331291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.331267    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.331508 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.331482    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.331641 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.331497    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.331704 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.331679    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.333844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.333819    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.334038 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.334022    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.394515 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.394479    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.394744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.394525    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhn5r\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-kube-api-access-zhn5r\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.430070 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.430041    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:41.484707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.484598    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" event={"ID":"d8276d1e-3e79-4fa2-8688-14f3da5c3a92","Type":"ContainerStarted","Data":"cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84"}
Nov 02 01:47:41.484707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.484675    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" event={"ID":"d8276d1e-3e79-4fa2-8688-14f3da5c3a92","Type":"ContainerStarted","Data":"7497cfedf2565eb8f5d00ea6d50e46ee734285837c7b6686f00ebea83ac1455c"}
Nov 02 01:47:41.487355 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.487328    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:47:41.487506 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.487383    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 01:47:41.497519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.497487    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:41.608886 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.608822    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podStartSLOduration=1.608802865 podStartE2EDuration="1.608802865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:41.604727055 +0000 UTC m=+1369.478368109" watchObservedRunningTime="2026-11-02 01:47:41.608802865 +0000 UTC m=+1369.482443921"
Nov 02 01:47:41.898796 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.898763    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"]
Nov 02 01:47:41.898952 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:47:41.898833    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f19908d_04f8_48a2_ab46_d3f6e9ff53fd.slice/crio-6ef47a111454844f48cd414d26258117b8245914bdeeada89c059e418d798a89 WatchSource:0}: Error finding container 6ef47a111454844f48cd414d26258117b8245914bdeeada89c059e418d798a89: Status 404 returned error can't find the container with id 6ef47a111454844f48cd414d26258117b8245914bdeeada89c059e418d798a89
Nov 02 01:47:41.901733 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.901699    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:41.901861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.901778    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:41.901861 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:41.901821    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:47:42.492489 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:42.492039    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:47:42.492489 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:42.492102    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 01:47:42.503334 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:42.503292    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:47:42.503531 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:42.503350    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 01:47:42.506924 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:42.506892    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" event={"ID":"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd","Type":"ContainerStarted","Data":"3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e"}
Nov 02 01:47:42.507065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:42.506955    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" event={"ID":"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd","Type":"ContainerStarted","Data":"6ef47a111454844f48cd414d26258117b8245914bdeeada89c059e418d798a89"}
Nov 02 01:47:42.690633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:42.681178    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podStartSLOduration=1.681156847 podStartE2EDuration="1.681156847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:42.673922222 +0000 UTC m=+1370.547563288" watchObservedRunningTime="2026-11-02 01:47:42.681156847 +0000 UTC m=+1370.554797900"
Nov 02 01:47:43.431116 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:43.431076    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:43.432565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:43.432538    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:47:43.432693 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:43.432588    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 01:47:43.492632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:43.488491    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:47:43.492632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:43.488555    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 01:47:43.499070 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:43.498965    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:47:43.499070 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:43.499018    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 01:47:44.431485 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:44.431447    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:47:44.431690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:44.431503    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 01:47:44.496633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:44.496470    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:47:44.496633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:44.496528    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 01:47:44.504663 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:44.500953    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:47:44.504663 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:44.501005    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 01:47:45.430940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:45.430906    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:47:45.431190 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:45.431168    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 01:47:45.497634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:45.492815    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:47:45.497634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:45.492893    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 01:47:45.507633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:45.502062    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:47:45.507633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:45.502134    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 01:47:46.132289 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:47:46.132252    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:47:46.430552 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:46.430451    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:47:46.430552 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:46.430497    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 01:47:46.492634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:46.491718    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 02 01:47:46.492634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:46.491793    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 02 01:47:46.507885 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:46.503729    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:47:46.507885 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:46.503814    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 01:47:47.431422 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:47.431381    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:47:47.431656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:47.431433    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 01:47:47.498331 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:47.498294    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:47:47.498560 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:47.498541    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 01:47:48.036629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:48.030340    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:48.036629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:48.031924    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:48.037161 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:48.036898    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 01:47:48.693636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:48.621441    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 01:47:48.693636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:48.621508    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 01:47:48.693636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:48.621717    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:47:48.693636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:48.621787    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 01:47:49.378635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:49.374711    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.447635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:49.440415    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 02 01:47:49.447635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:49.440477    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 02 01:47:49.608783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:49.608737    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.777646 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:49.772703    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.930761 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:49.930715    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:49.975277 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:49.975221    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.015933 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.015869    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.049038 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.048903    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.261855 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.261777    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.674769 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.674711    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:50.710978 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.710933    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.711994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.711960    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.715966 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.712472    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:47:50.907750 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.907703    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.942126 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.941972    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:50.981543 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:50.981499    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:47:51.241686 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.238168    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 01:47:51.241686 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.238631    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:47:51.302790 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:47:51.301397    4287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.036s"
Nov 02 01:47:51.303116 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.303095    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:47:51.303291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.303278    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:47:51.303421 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.303409    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:47:51.303581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.303569    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:47:51.303748 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.303734    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:47:51.313591 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.313330    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.316394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.316371    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:47:51.316715 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.316700    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:47:51.506775 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.506678    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Nov 02 01:47:51.540633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.540189    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:51.540633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.540267    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.616779 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.616749    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:47:51.718726 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.718689    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:51.743724 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.743684    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:51.744211 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.730131    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:47:51.744628 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:51.721465    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:47:52.125660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.116974    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:47:52.135882 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.135397    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:47:52.159547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.154926    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:52.168227 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.168200    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:52.177987 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.177960    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:47:52.240293 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.239675    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:52.326570 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.326541    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:47:52.786770 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.786736    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:47:52.787155 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.786787    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:47:52.903426 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.903392    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:52.903868 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.903836    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:47:52.904547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:47:52.904522    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 01:48:09.413795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.413758    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"]
Nov 02 01:48:09.419024 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.419008    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.496744 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.496704    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.496932 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.496764    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.497007 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.496928    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4245e-60d6-4175-985a-5471f6fa790e-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.497007 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.496972    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.497081 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.497049    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9v9\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-kube-api-access-9w9v9\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.497144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.497123    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.497254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.497168    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.497254 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.497203    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.497333 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.497260    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4245e-60d6-4175-985a-5471f6fa790e-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.526332 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.526285    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"]
Nov 02 01:48:09.597985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.597956    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4245e-60d6-4175-985a-5471f6fa790e-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598178 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.597991    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598178 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598014    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9v9\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-kube-api-access-9w9v9\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598178 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598049    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598178 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598081    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598178 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598106    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598467 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598240    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4245e-60d6-4175-985a-5471f6fa790e-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598467 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598403    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598467 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598413    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598467 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598454    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598663 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598477    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598663 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598501    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598780    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.598935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.598921    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4245e-60d6-4175-985a-5471f6fa790e-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.600781 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.600759    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.601031 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.601011    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4245e-60d6-4175-985a-5471f6fa790e-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.661026 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.660998    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.661142 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.661071    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9v9\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-kube-api-access-9w9v9\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.733244 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.733183    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:09.962020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.961965    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"]
Nov 02 01:48:09.966771 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:09.966745    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.027222 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.027194    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"]
Nov 02 01:48:10.107953 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.107912    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.107953 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.107943    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162987e-acf2-4901-8631-822c903e32d8-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.108202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.107991    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.108202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.108082    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.108202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.108175    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.108378 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.108310    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162987e-acf2-4901-8631-822c903e32d8-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.108378 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.108355    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.108483 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.108420    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftshq\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-kube-api-access-ftshq\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.108541 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.108502    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.190934 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.190901    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"]
Nov 02 01:48:10.191256 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:48:10.191233    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e4245e_60d6_4175_985a_5471f6fa790e.slice/crio-071b74d116f9d4a00d3f929cc0f05358c0fe125b2b6a9c02d1d772e5f9e91e5b WatchSource:0}: Error finding container 071b74d116f9d4a00d3f929cc0f05358c0fe125b2b6a9c02d1d772e5f9e91e5b: Status 404 returned error can't find the container with id 071b74d116f9d4a00d3f929cc0f05358c0fe125b2b6a9c02d1d772e5f9e91e5b
Nov 02 01:48:10.193574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.193540    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:10.193697 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.193634    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:10.193776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.193708    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:10.209519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209499    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162987e-acf2-4901-8631-822c903e32d8-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209536    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209558    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftshq\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-kube-api-access-ftshq\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209728 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209619    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209728 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209696    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209728 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209720    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162987e-acf2-4901-8631-822c903e32d8-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209778    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209803    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209832    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.209935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209912    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.210144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.209983    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.210206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.210167    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.210433 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.210346    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.210543 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.210523    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162987e-acf2-4901-8631-822c903e32d8-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.214173 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.212250    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.214173 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.212419    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162987e-acf2-4901-8631-822c903e32d8-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.270221 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.270194    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.270455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.270436    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftshq\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-kube-api-access-ftshq\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.280818 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.280753    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:10.438246 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.438214    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"]
Nov 02 01:48:10.443357 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.443328    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" event={"ID":"84e4245e-60d6-4175-985a-5471f6fa790e","Type":"ContainerStarted","Data":"f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802"}
Nov 02 01:48:10.443493 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.443369    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" event={"ID":"84e4245e-60d6-4175-985a-5471f6fa790e","Type":"ContainerStarted","Data":"071b74d116f9d4a00d3f929cc0f05358c0fe125b2b6a9c02d1d772e5f9e91e5b"}
Nov 02 01:48:10.443493 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.443455    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.507218 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.507182    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"]
Nov 02 01:48:10.514165 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514130    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.514347 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514183    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.514434 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514344    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.514513 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514497    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phm46\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-kube-api-access-phm46\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.514570 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514556    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.514668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514640    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec77c94b-73bd-4608-be47-ec0997c04d8b-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.514731 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514721    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.514790 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514748    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.514790 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.514781    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.571470 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.571362    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podStartSLOduration=1.571344045 podStartE2EDuration="1.571344045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:10.570432853 +0000 UTC m=+1398.444073905" watchObservedRunningTime="2026-11-02 01:48:10.571344045 +0000 UTC m=+1398.444985098"
Nov 02 01:48:10.615912 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.615882    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phm46\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-kube-api-access-phm46\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.615939    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.615981    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec77c94b-73bd-4608-be47-ec0997c04d8b-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616046    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616067    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616363 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616093    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616451 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616431    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616511 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616448    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616511 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616472    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616511 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616501    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616541    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.616973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.616949    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.617253 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.617231    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec77c94b-73bd-4608-be47-ec0997c04d8b-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.617394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.617378    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.619457 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.619435    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.619859 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.619842    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.668687 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.668650    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"]
Nov 02 01:48:10.669032 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:48:10.669008    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode162987e_acf2_4901_8631_822c903e32d8.slice/crio-1f74fd66b6747416397fea00a0677ef30338b2461440bb81f788b04737c5bfd1 WatchSource:0}: Error finding container 1f74fd66b6747416397fea00a0677ef30338b2461440bb81f788b04737c5bfd1: Status 404 returned error can't find the container with id 1f74fd66b6747416397fea00a0677ef30338b2461440bb81f788b04737c5bfd1
Nov 02 01:48:10.671561 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.671532    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:10.671634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.671594    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:10.671692 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.671676    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:10.679329 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.679303    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phm46\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-kube-api-access-phm46\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.679435 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.679314    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.700795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.700757    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"]
Nov 02 01:48:10.706350 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.706317    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.733748 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.733719    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:10.758116 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.758072    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:10.772633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.771781    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"]
Nov 02 01:48:10.821656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.821520    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.821656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.821586    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r259r\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-kube-api-access-r259r\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.822016 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.821706    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.822016 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.821745    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.822016 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.821833    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.822016 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.821861    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.822016 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.821925    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.822016 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.821970    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.822016 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.822007    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923272    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923325    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923354    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923389    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923425    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r259r\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-kube-api-access-r259r\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923482    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923507    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923567    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923585    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.923983    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.924450    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.924643    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.925254    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.926680 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.925503    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.930573 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.928981    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.933637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.933571    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.993902 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.993412    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:10.995998 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.995964    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"]
Nov 02 01:48:10.998359 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:10.998292    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r259r\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-kube-api-access-r259r\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:11.003040 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.003023    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"]
Nov 02 01:48:11.003239 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.003199    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.007930 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.007912    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.020628 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.020576    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"]
Nov 02 01:48:11.072149 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.070894    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:11.090757 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.090726    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"]
Nov 02 01:48:11.126377 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126300    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.126377 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126356    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.126658 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126486    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ab77-e085-41b1-b594-44a22aa6b640-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.126658 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126582    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrhg\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-kube-api-access-9rrhg\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.126658 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126628    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.126806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126680    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.126806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126727    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4058d510-3481-400b-bb9a-6c9454ff1cef-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.126806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126759    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.126806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126781    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.126964 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126803    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.126964 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126835    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.126964 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126894    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.126964 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126947    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.127095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.126978    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.127095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.127016    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.127095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.127064    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szf2z\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-kube-api-access-szf2z\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.127212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.127101    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ab77-e085-41b1-b594-44a22aa6b640-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.127212 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.127131    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.229046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229004    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.229239 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229072    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.229332 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229264    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ab77-e085-41b1-b594-44a22aa6b640-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.229398 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229376    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrhg\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-kube-api-access-9rrhg\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.229456 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229432    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.229514 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229481    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.229574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229514    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.229648 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229581    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4058d510-3481-400b-bb9a-6c9454ff1cef-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.229712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229644    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.229712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229670    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.229831 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229716    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.229831 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229749    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.229831 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229825    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.229985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229905    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.229985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.229956    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.230095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.230000    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.230095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.230070    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szf2z\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-kube-api-access-szf2z\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.230204 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.230125    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ab77-e085-41b1-b594-44a22aa6b640-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.230204 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.230157    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.230306 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.230275    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.230642 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.230585    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.230885 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.230845    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.231115 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.231095    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.231216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.231180    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.231996 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.231971    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ab77-e085-41b1-b594-44a22aa6b640-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.232090 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.232024    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.232267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.232247    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.234138 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.233953    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4058d510-3481-400b-bb9a-6c9454ff1cef-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.235207 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.235161    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.237426 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.237352    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.238024 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.237972    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ab77-e085-41b1-b594-44a22aa6b640-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.245828 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.245561    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.271842 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.261721    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"]
Nov 02 01:48:11.279191 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:48:11.277991    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec77c94b_73bd_4608_be47_ec0997c04d8b.slice/crio-be09017a03b6b5f18a80f6b61a5690cb27ed241f91d8fcd602848b9ed27ffe04 WatchSource:0}: Error finding container be09017a03b6b5f18a80f6b61a5690cb27ed241f91d8fcd602848b9ed27ffe04: Status 404 returned error can't find the container with id be09017a03b6b5f18a80f6b61a5690cb27ed241f91d8fcd602848b9ed27ffe04
Nov 02 01:48:11.284494 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.283724    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:11.284494 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.283804    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:11.284494 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.283847    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:11.299417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.298881    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.300267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.300239    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrhg\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-kube-api-access-9rrhg\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.300418 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.300381    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.301232 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.301208    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szf2z\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-kube-api-access-szf2z\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.325724 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.325653    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:11.333576 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.333553    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:11.601227 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.551307    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"]
Nov 02 01:48:11.601815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.601659    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" event={"ID":"e162987e-acf2-4901-8631-822c903e32d8","Type":"ContainerStarted","Data":"4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488"}
Nov 02 01:48:11.601815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.601744    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" event={"ID":"e162987e-acf2-4901-8631-822c903e32d8","Type":"ContainerStarted","Data":"1f74fd66b6747416397fea00a0677ef30338b2461440bb81f788b04737c5bfd1"}
Nov 02 01:48:11.659533 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.648514    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:11.659533 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.648584    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:11.698916 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.698879    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" event={"ID":"ec77c94b-73bd-4608-be47-ec0997c04d8b","Type":"ContainerStarted","Data":"34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085"}
Nov 02 01:48:11.699174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.699156    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" event={"ID":"ec77c94b-73bd-4608-be47-ec0997c04d8b","Type":"ContainerStarted","Data":"be09017a03b6b5f18a80f6b61a5690cb27ed241f91d8fcd602848b9ed27ffe04"}
Nov 02 01:48:11.699374 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.699351    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:11.735479 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.735456    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:11.735629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.735591    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:11.759850 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.759772    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:11.811424 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.811365    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podStartSLOduration=2.811346688 podStartE2EDuration="2.811346688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:11.786095764 +0000 UTC m=+1399.659736842" watchObservedRunningTime="2026-11-02 01:48:11.811346688 +0000 UTC m=+1399.684987741"
Nov 02 01:48:11.816396 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.816361    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"]
Nov 02 01:48:11.834795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.834686    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:11.834795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.834753    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:11.835291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:11.834949    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:12.026656 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:48:12.010305    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4058d510_3481_400b_bb9a_6c9454ff1cef.slice/crio-401a7668ac3a71c4d801e57e32c433a4946e787023b6cd1b5aeab111907212e1 WatchSource:0}: Error finding container 401a7668ac3a71c4d801e57e32c433a4946e787023b6cd1b5aeab111907212e1: Status 404 returned error can't find the container with id 401a7668ac3a71c4d801e57e32c433a4946e787023b6cd1b5aeab111907212e1
Nov 02 01:48:12.031685 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.028796    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:12.031685 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.028895    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:12.031685 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.028955    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:12.040745 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.037735    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podStartSLOduration=2.037715969 podStartE2EDuration="2.037715969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:11.92398853 +0000 UTC m=+1399.797629583" watchObservedRunningTime="2026-11-02 01:48:12.037715969 +0000 UTC m=+1399.911357022"
Nov 02 01:48:12.040745 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.040053    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"]
Nov 02 01:48:12.288634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.284949    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:12.302632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.297958    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:12.302632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.298036    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:12.738312 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.734728    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:12.738312 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.734781    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:12.755071 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.755033    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" event={"ID":"4058d510-3481-400b-bb9a-6c9454ff1cef","Type":"ContainerStarted","Data":"be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49"}
Nov 02 01:48:12.758911 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.758880    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" event={"ID":"4058d510-3481-400b-bb9a-6c9454ff1cef","Type":"ContainerStarted","Data":"401a7668ac3a71c4d801e57e32c433a4946e787023b6cd1b5aeab111907212e1"}
Nov 02 01:48:12.762518 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.762491    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:12.762794 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.762769    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:12.766464 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.765550    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" event={"ID":"7862ab77-e085-41b1-b594-44a22aa6b640","Type":"ContainerStarted","Data":"51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1"}
Nov 02 01:48:12.766464 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.765626    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" event={"ID":"7862ab77-e085-41b1-b594-44a22aa6b640","Type":"ContainerStarted","Data":"c869aaf8d050f768735c791f519236fc4f73e74ae53032c2dc8b2ce94983ef6d"}
Nov 02 01:48:12.785642 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.782920    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" event={"ID":"0bc7e3ec-0c80-45a6-9bae-03108eb0b873","Type":"ContainerStarted","Data":"fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a"}
Nov 02 01:48:12.785642 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:12.782978    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" event={"ID":"0bc7e3ec-0c80-45a6-9bae-03108eb0b873","Type":"ContainerStarted","Data":"020ab868259b5b015a6f8c258389cdca33155d82aa903445a501564422883aa5"}
Nov 02 01:48:13.032734 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.031259    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podStartSLOduration=3.03123526 podStartE2EDuration="3.03123526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:12.889181453 +0000 UTC m=+1400.762822508" watchObservedRunningTime="2026-11-02 01:48:13.03123526 +0000 UTC m=+1400.904876313"
Nov 02 01:48:13.102637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.089682    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:13.102637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.094732    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.102637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.094788    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:13.202635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.198021    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podStartSLOduration=3.198000379 podStartE2EDuration="3.198000379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:13.047136804 +0000 UTC m=+1400.920777857" watchObservedRunningTime="2026-11-02 01:48:13.198000379 +0000 UTC m=+1401.071641431"
Nov 02 01:48:13.202635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.198187    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podStartSLOduration=3.198181022 podStartE2EDuration="3.198181022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:13.195245128 +0000 UTC m=+1401.068886182" watchObservedRunningTime="2026-11-02 01:48:13.198181022 +0000 UTC m=+1401.071822075"
Nov 02 01:48:13.286692 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.282991    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.286692 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.283064    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:13.331649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.329686    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:13.331649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.329827    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.331649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.329868    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:13.347783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.346639    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:13.347783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.346780    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.347783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.346824    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:13.586922 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.586785    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.622771 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.622719    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.628783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.628741    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.729649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.725709    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.737803 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.737762    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.738119 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.738091    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:13.747318 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.747271    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:13.766069 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.766027    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:13.766341 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:13.766297    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:14.085123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.085084    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.085447 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.085403    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:14.147751 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.147708    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:48:14.249804 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.249425    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:14.283048 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.283011    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.283339 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.283306    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:14.325969 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.325933    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.326229 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.326204    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:14.336062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.335971    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.336294 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.336266    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:14.636205 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.636081    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 01:48:14.636768 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.636743    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:48:14.706788 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.706733    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.716771 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.716734    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:48:14.729769 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.729724    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.730547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.730518    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:48:14.745222 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.745186    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.745503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.745473    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:14.751723 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.751682    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:14.766798 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.766746    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:14.767277 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.767229    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:14.767390 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.767280    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:14.767459 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:14.767410    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:48:15.225662 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.211312    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.273724 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.243170    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:48:15.273724 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.268585    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.273724 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.270359    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:48:15.448001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.447941    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:48:15.473682 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.468924    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.473682 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.468988    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:15.524030 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.522371    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.524030 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.522429    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:15.524030 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.522515    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.524030 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.522538    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:15.536463 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.536315    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.536463 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.536373    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:15.637756 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.637709    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.721006 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.720959    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.731451 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.731406    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:15.737428 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.737395    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.737714 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.737677    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:15.775810 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.775709    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:15.776508 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.776420    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:15.776719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.776466    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:15.870633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:15.865509    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:48:16.081226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.081023    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.081226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.081095    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:16.280774 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.280723    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:16.334745 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.316099    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:16.341278 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.335083    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.341569 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.341522    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:16.341759 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.335547    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.341878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.341857    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:16.341997 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.335584    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.352124 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.352092    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:16.478684 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.454699    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:16.688686 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.612383    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:48:16.692815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.652836    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 01:48:16.733325 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.733283    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 01:48:16.752635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.736455    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.752635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.740955    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:16.768715 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.762652    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:16.768715 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.762705    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:16.796576 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:16.786990    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:17.109517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.109481    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.140042 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.139982    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:17.316139 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.315768    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.354664 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:48:17.335035    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:48:17.354664 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.335500    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.354664 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.335547    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:17.354664 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.335639    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.354664 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.335663    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:17.354664 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.335714    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:17.522708 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.522671    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.577346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.549423    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:48:17.600316 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.577191    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:17.723534 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.723490    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.739639 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.735687    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.740047 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.740021    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.789433 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.789338    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:17.790732 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.789202    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:17.793640 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.790911    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:17.802813 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.802748    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:17.837193 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:17.804559    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.093919 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.093824    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.118682 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.118589    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:18.395693 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.369323    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.395693 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.387062    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.395693 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.387138    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:18.395693 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.387184    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.409176 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.404317    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.409176 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.404375    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:18.501702 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.435344    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.501702 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.435417    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:18.593882 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.593818    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.785829 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.770711    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:48:18.785829 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.771170    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:48:18.815780 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.804706    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.815780 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.805103    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:18.815780 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.805553    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 01:48:18.828579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.826226    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.828579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.826278    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:18.828579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.826368    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:18.828579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:18.826390    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:19.079121 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.078949    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Nov 02 01:48:19.140482 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.139089    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.140482 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.139151    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:19.291767 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.291707    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.301836 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.301772    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:19.584635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.584519    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:48:19.597636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.596053    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:19.597836 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.597722    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:19.609644 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.599140    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 02 01:48:19.638634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.632752    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.638634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.632815    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:19.638634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.632927    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.638634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.632953    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:19.779634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.758302    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.779634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.758393    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 02 01:48:19.815637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.814205    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:19.815637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.814243    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 01:48:19.815637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.814289    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:19.815637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.814408    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 02 01:48:19.815637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.814417    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:19.815637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.814485    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:19.872720 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:19.846773    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:20.186695 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.137042    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:20.335637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.313146    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:20.384645 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.355814    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:20.384645 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.355888    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:20.384645 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.356021    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:20.384645 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.356092    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:20.384645 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.373525    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:20.438737 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.426526    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:20.595804 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.595752    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:20.596361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.596325    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:20.623737 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.623689    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:20.773970 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.773934    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:20.774267 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.774239    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:20.816775 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.816734    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:20.818825 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.818793    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:20.819317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.818998    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:20.848585 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.848533    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:20.877746 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:20.877703    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 02 01:48:21.080469 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.080365    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:21.080776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.080721    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:21.308635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.307886    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:21.309161 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.308953    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:21.309161 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.309002    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:21.326776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.326744    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:21.327009 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.326983    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:21.356175 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.356079    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:21.356409 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.356384    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:21.604640 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.602709    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:21.607735 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.606676    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:48:21.670746 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.667522    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:21.826515 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.817540    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:21.826515 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.817601    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:21.894444 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.883380    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:21.894444 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.884161    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 01:48:21.894444 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.884316    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:21.894444 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:21.884640    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:48:22.052843 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.052804    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:22.053147 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.053105    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:22.208313 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.208223    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:22.208574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.208545    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:22.317164 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.317131    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 02 01:48:22.337247 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.337202    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 02 01:48:22.374914 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.374876    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:22.379640 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.375136    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:22.379964 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.379938    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:22.389722 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.389682    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:22.626871 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.626820    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:22.635003 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.634942    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:22.721678 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.695424    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:22.796638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.793671    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 02 01:48:22.796638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.793727    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 02 01:48:22.884854 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.884736    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:22.885927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.885891    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:22.886196 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.886147    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:22.886432 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.886127    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:22.915036 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.898224    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:22.915036 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:22.907954    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:23.020713 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.020668    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:48:23.102643 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.099100    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:23.102952 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.102900    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:23.391912 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.391873    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:23.392214 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.392187    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:23.465499 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.465463    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:23.465792 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.465764    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:23.471931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.471901    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:48:23.568263 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.568233    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:48:23.588672 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.581724    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:48:23.628698 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.627860    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:23.633117 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.633079    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:48:23.634133 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.634103    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:23.643677 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.641132    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:48:23.804877 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.804845    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:48:23.816642 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.808685    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:48:23.859954 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.859917    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:23.864046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.864024    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:23.899047 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:23.898965    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 01:48:24.089119 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.086357    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 01:48:24.089529 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.089476    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 01:48:24.184438 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.184357    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:24.185321 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.185288    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:24.186551 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.186510    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:48:24.326103 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.326074    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 01:48:24.326283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.326126    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 01:48:24.334444 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.334415    4287 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 02 01:48:24.334557 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:24.334450    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 02 01:48:25.272635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.271762    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:25.274650 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.273518    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:25.274650 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.274549    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:48:25.349285 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.349248    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:25.350558 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.350530    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:25.351323 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.351303    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:48:25.494779 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.494746    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:25.495509 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.495486    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:25.496714 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:25.496680    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 01:48:28.789867 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:28.789818    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:28.792577 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:28.792525    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:28.888550 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:28.888506    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.106813 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.106706    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.336735 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.336690    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:29.374866 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.374705    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.590635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.589597    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.601500 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.600717    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.650645 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.650519    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.770830 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.770781    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.799926 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.797824    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:29.826658 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.797955    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 01:48:29.869636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.827564    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:48:30.151370 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:30.151338    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:48:30.264895 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:29.894243    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:30.376044 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:30.375997    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:30.601377 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:30.601334    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:30.651635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:30.603735    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:30.890690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:30.775174    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 01:48:30.890690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:30.804728    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:30.890690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:30.868820    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 01:48:30.918705 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:30.893529    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.218375 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.218277    4287 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:31.218702 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.218666    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.306801 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.306748    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Nov 02 01:48:31.408635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.402736    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.594634 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:48:31.569144    4287 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.314s"
Nov 02 01:48:31.787547 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.787513    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:48:31.787892 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.787873    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:48:31.828894 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.828867    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:48:31.829262 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.829244    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:48:31.830878 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.697044    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Nov 02 01:48:31.831882 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.725966    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.832647 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.807599    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.844503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.790677    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:31.844503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.833116    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:31.844503 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.784845    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 01:48:31.860791 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.860707    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:48:31.860992 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.860980    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:48:31.874504 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:31.874463    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:32.420982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:32.420939    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:48:32.593630 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:32.571314    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:48:32.596788 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:32.596763    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:48:32.651917 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:32.651887    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 01:48:32.844760 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:32.844713    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:32.858129 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:32.846395    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:48:32.858129 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:32.848696    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:48:32.858129 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:32.848736    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:48:33.421645 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:33.404418    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:48:33.473282 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:33.472425    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 01:48:33.531486 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:33.531456    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:48:39.652841 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:39.652803    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7048-2b3bf4"]
Nov 02 01:48:39.657147 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:39.657128    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7048-2b3bf4"
Nov 02 01:48:39.710038 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:39.710009    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7048-2b3bf4"]
Nov 02 01:48:39.755441 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:39.755412    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n9czn\""
Nov 02 01:48:39.761323 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:39.761289    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56g4\" (UniqueName: \"kubernetes.io/projected/04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4-kube-api-access-c56g4\") pod \"test-curl-7048-2b3bf4\" (UID: \"04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4\") " pod="opendatahub/test-curl-7048-2b3bf4"
Nov 02 01:48:39.862215 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:39.862184    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56g4\" (UniqueName: \"kubernetes.io/projected/04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4-kube-api-access-c56g4\") pod \"test-curl-7048-2b3bf4\" (UID: \"04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4\") " pod="opendatahub/test-curl-7048-2b3bf4"
Nov 02 01:48:39.923504 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:39.923437    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56g4\" (UniqueName: \"kubernetes.io/projected/04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4-kube-api-access-c56g4\") pod \"test-curl-7048-2b3bf4\" (UID: \"04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4\") " pod="opendatahub/test-curl-7048-2b3bf4"
Nov 02 01:48:39.970536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:39.970502    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7048-2b3bf4"
Nov 02 01:48:40.363642 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:40.363595    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7048-2b3bf4"]
Nov 02 01:48:40.365139 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:48:40.365106    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04248cc9_7b06_4ba0_b2d3_3c2d8b2c1ed4.slice/crio-5447f9ccb8d02f61ea79c18d319539d5a41e44350f424469cb74175f9c56b648 WatchSource:0}: Error finding container 5447f9ccb8d02f61ea79c18d319539d5a41e44350f424469cb74175f9c56b648: Status 404 returned error can't find the container with id 5447f9ccb8d02f61ea79c18d319539d5a41e44350f424469cb74175f9c56b648
Nov 02 01:48:40.367100 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:40.367079    4287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:48:40.531619 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:40.531560    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7048-2b3bf4" event={"ID":"04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4","Type":"ContainerStarted","Data":"5447f9ccb8d02f61ea79c18d319539d5a41e44350f424469cb74175f9c56b648"}
Nov 02 01:48:43.553681 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:43.553642    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7048-2b3bf4" event={"ID":"04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4","Type":"ContainerStarted","Data":"5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d"}
Nov 02 01:48:43.667124 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:43.667061    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7048-2b3bf4" podStartSLOduration=2.135895523 podStartE2EDuration="4.667044198s" totalImagesPullingTime="2.531148675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:43.665694704 +0000 UTC m=+1431.539335758" watchObservedRunningTime="2026-11-02 01:48:43.667044198 +0000 UTC m=+1431.540685251"
Nov 02 01:48:44.511306 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:44.511271    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"]
Nov 02 01:48:44.511524 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:44.511497    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy" containerID="cri-o://ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637" gracePeriod=30
Nov 02 01:48:45.295242 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:45.295206    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7048-2b3bf4"]
Nov 02 01:48:45.299780 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:45.299749    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7048-2b3bf4"]
Nov 02 01:48:45.566532 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:45.566341    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7048-2b3bf4" podUID="04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4" containerName="test-curl-7048-2b3bf4" containerID="cri-o://5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d" gracePeriod=2
Nov 02 01:48:45.617211 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:48:45.617166    4287 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7048-2b3bf4\" is forbidden: User \"system:node:ip-10-0-134-183.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-183.us-west-1.compute.internal' and this object" podUID="04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4" pod="opendatahub/test-curl-7048-2b3bf4"
Nov 02 01:48:46.038410 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:48:46.038375    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:48:47.782672 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:47.782645    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7048-2b3bf4"
Nov 02 01:48:47.881265 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:47.881179    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4-kube-api-access-c56g4\" (UniqueName: \"kubernetes.io/projected/04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4-kube-api-access-c56g4\") pod \"04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4\" (UID: \"04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4\") "
Nov 02 01:48:47.883722 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:47.883695    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4-kube-api-access-c56g4" pod "04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4" (UID: "04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4"). InnerVolumeSpecName "kube-api-access-c56g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:47.983081 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:47.983047    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c56g4\" (UniqueName: \"kubernetes.io/projected/04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4-kube-api-access-c56g4\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:48.238139 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:48.238057    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4" path="/var/lib/kubelet/pods/04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4/volumes"
Nov 02 01:48:48.595108 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:48.595008    4287 generic.go:356] "Generic (PLEG): container finished" podID="04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4" containerID="5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d" exitCode=137
Nov 02 01:48:48.595630 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:48.595589    4287 scope.go:122] "RemoveContainer" containerID="5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d"
Nov 02 01:48:48.595976 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:48.595955    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7048-2b3bf4"
Nov 02 01:48:48.616931 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:48.616902    4287 scope.go:122] "RemoveContainer" containerID="5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d"
Nov 02 01:48:48.617422 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:48:48.617362    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d\": container with ID starting with 5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d not found: ID does not exist" containerID="5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d"
Nov 02 01:48:48.617556 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:48.617433    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d"} err="failed to get container status \"5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d\": rpc error: code = NotFound desc = could not find container \"5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d\": container with ID starting with 5d42b10fea3ed9b62752b83bcf2c8f45b725bf0fc4a7b5c3173873fa2e83c80d not found: ID does not exist"
Nov 02 01:48:49.832906 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.832865    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:48:49.934602 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.934557    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-token\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-token\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.934845 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.934667    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-envoy\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.935138 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.935109    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-credential-socket\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.935290 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.935165    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-data\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.935290 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.935209    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-podinfo\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.935972 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.935932    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-certs" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:49.935972 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.935948    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-credential-socket" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:49.936430 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.936407    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-data" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:49.936518 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.935264    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-certs\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.936518 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.936490    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-kube-api-access-bqbgt\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-kube-api-access-bqbgt\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.936657 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.936554    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-socket\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.936657 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.936596    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istiod-ca-cert\") pod \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\" (UID: \"d1071dc1-e1dc-4ad1-ae82-91a80646a631\") "
Nov 02 01:48:49.938040 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.937371    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:49.938040 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.937400    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:49.938040 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.937415    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:49.938283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.938198    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-socket" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:49.938371 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.938341    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-token" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:49.938486 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.938474    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istiod-ca-cert" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:48:49.940457 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.940430    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-envoy" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:48:49.940891 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.940863    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-podinfo" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:48:49.940995 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:49.940913    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-kube-api-access-bqbgt" pod "d1071dc1-e1dc-4ad1-ae82-91a80646a631" (UID: "d1071dc1-e1dc-4ad1-ae82-91a80646a631"). InnerVolumeSpecName "kube-api-access-bqbgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:48:50.041669 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.039040    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:50.041669 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.039080    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:50.041669 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.039096    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:50.041669 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.039110    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:50.041669 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.039124    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1071dc1-e1dc-4ad1-ae82-91a80646a631-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:50.041669 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.039137    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqbgt\" (UniqueName: \"kubernetes.io/projected/d1071dc1-e1dc-4ad1-ae82-91a80646a631-kube-api-access-bqbgt\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:48:50.615899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.615538    4287 generic.go:356] "Generic (PLEG): container finished" podID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerID="ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637" exitCode=0
Nov 02 01:48:50.615899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.615601    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"
Nov 02 01:48:50.615899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.615760    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" event={"ID":"d1071dc1-e1dc-4ad1-ae82-91a80646a631","Type":"ContainerDied","Data":"ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637"}
Nov 02 01:48:50.615899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.615824    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2" event={"ID":"d1071dc1-e1dc-4ad1-ae82-91a80646a631","Type":"ContainerDied","Data":"ac9f6d2d69d9dec4370bca2595c6b417f92b4c8c7b6ee372256ccf9795def9d4"}
Nov 02 01:48:50.615899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.615850    4287 scope.go:122] "RemoveContainer" containerID="ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637"
Nov 02 01:48:50.628679 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.628658    4287 scope.go:122] "RemoveContainer" containerID="ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a"
Nov 02 01:48:50.641222 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.641188    4287 scope.go:122] "RemoveContainer" containerID="ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637"
Nov 02 01:48:50.641622 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:48:50.641576    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637\": container with ID starting with ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637 not found: ID does not exist" containerID="ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637"
Nov 02 01:48:50.641740 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.641639    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637"} err="failed to get container status \"ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637\": rpc error: code = NotFound desc = could not find container \"ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637\": container with ID starting with ad91d8c269567780d2673349bcf2579f9d96bcfb1c09839039758ae098f97637 not found: ID does not exist"
Nov 02 01:48:50.641740 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.641666    4287 scope.go:122] "RemoveContainer" containerID="ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a"
Nov 02 01:48:50.642021 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:48:50.642000    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a\": container with ID starting with ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a not found: ID does not exist" containerID="ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a"
Nov 02 01:48:50.642101 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.642040    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a"} err="failed to get container status \"ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a\": rpc error: code = NotFound desc = could not find container \"ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a\": container with ID starting with ec6ddec90b430ae72de4b05efb638077c17e725518e049a185d33174594a019a not found: ID does not exist"
Nov 02 01:48:50.787714 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.787681    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"]
Nov 02 01:48:50.792332 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:50.792302    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-98dt2"]
Nov 02 01:48:51.889076 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.888243    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"]
Nov 02 01:48:51.889956 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.889931    4287 cpu_manager.go:414] "containerMap: removing container" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy"
Nov 02 01:48:51.889956 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.889957    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy"
Nov 02 01:48:51.890145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.889984    4287 cpu_manager.go:414] "containerMap: removing container" podUID="04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4" containerName="test-curl-7048-2b3bf4"
Nov 02 01:48:51.890145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.889992    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4" containerName="test-curl-7048-2b3bf4"
Nov 02 01:48:51.890145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.890013    4287 cpu_manager.go:414] "containerMap: removing container" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy"
Nov 02 01:48:51.890145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.890020    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy"
Nov 02 01:48:51.890360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.890166    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy"
Nov 02 01:48:51.890360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.890181    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" containerName="istio-proxy"
Nov 02 01:48:51.890360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.890188    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="04248cc9-7b06-4ba0-b2d3-3c2d8b2c1ed4" containerName="test-curl-7048-2b3bf4"
Nov 02 01:48:51.895107 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.895085    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:51.969842 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:51.969803    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"]
Nov 02 01:48:52.022397 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.022365    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fd723454-openshift-default-dockercfg-z4cg8\""
Nov 02 01:48:52.077490 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.077441    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-certs\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.077490 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.077490    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-data\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.077769 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.077651    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-credential-socket\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.077769 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.077716    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m886z\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-kube-api-access-m886z\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.077850 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.077821    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-envoy\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.077921 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.077905    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-podinfo\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.078013 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.077998    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25048cc7-4e48-4699-a20c-5740ae3065e0-istiod-ca-cert\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.078073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.078058    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-socket\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.078147 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.078130    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-token\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.179261 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179173    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-certs\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.179261 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179222    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-data\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.179589 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179278    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-credential-socket\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.179589 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179325    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m886z\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-kube-api-access-m886z\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.179589 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179372    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-envoy\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.179589 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179436    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-podinfo\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.180472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179593    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25048cc7-4e48-4699-a20c-5740ae3065e0-istiod-ca-cert\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.180472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179671    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-socket\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.180472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179721    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-token\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.180472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179722    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-data\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.180472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.179815    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-credential-socket\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.180472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.180002    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-certs\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.180472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.180138    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-socket\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.180472 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.180429    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25048cc7-4e48-4699-a20c-5740ae3065e0-istiod-ca-cert\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.182774 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.182746    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-envoy\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.183015 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.182991    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-podinfo\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.239509 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.239472    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1071dc1-e1dc-4ad1-ae82-91a80646a631" path="/var/lib/kubelet/pods/d1071dc1-e1dc-4ad1-ae82-91a80646a631/volumes"
Nov 02 01:48:52.240471 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.240440    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-token\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.240575 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.240444    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m886z\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-kube-api-access-m886z\") pod \"e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") " pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:52.560278 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.560241    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fd723454-openshift-default-dockercfg-z4cg8\""
Nov 02 01:48:52.569501 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:52.569466    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:53.049101 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:53.049071    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"]
Nov 02 01:48:53.050463 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:48:53.050428    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25048cc7_4e48_4699_a20c_5740ae3065e0.slice/crio-7b43914da57a2bab1323112c168dc9a6a4bf17ef26bf0970f64adaddeee5e0ac WatchSource:0}: Error finding container 7b43914da57a2bab1323112c168dc9a6a4bf17ef26bf0970f64adaddeee5e0ac: Status 404 returned error can't find the container with id 7b43914da57a2bab1323112c168dc9a6a4bf17ef26bf0970f64adaddeee5e0ac
Nov 02 01:48:53.053338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:53.053303    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:53.053416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:53.053400    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:53.053465 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:53.053447    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:53.660977 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:53.660937    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" event={"ID":"25048cc7-4e48-4699-a20c-5740ae3065e0","Type":"ContainerStarted","Data":"f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9"}
Nov 02 01:48:53.661168 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:53.661004    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" event={"ID":"25048cc7-4e48-4699-a20c-5740ae3065e0","Type":"ContainerStarted","Data":"7b43914da57a2bab1323112c168dc9a6a4bf17ef26bf0970f64adaddeee5e0ac"}
Nov 02 01:48:53.784077 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:53.784019    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" podStartSLOduration=2.784003106 podStartE2EDuration="2.784003106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:53.780892427 +0000 UTC m=+1441.654533480" watchObservedRunningTime="2026-11-02 01:48:53.784003106 +0000 UTC m=+1441.657644157"
Nov 02 01:48:54.569769 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:54.569726    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:54.575749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:54.575720    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:54.671767 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:54.671727    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:54.672915 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:54.672888    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:48:55.336168 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.336131    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"]
Nov 02 01:48:55.340402 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.340383    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.442274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.442242    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"]
Nov 02 01:48:55.446158 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446124    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.446334 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446179    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.446432 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446405    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.446676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446451    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.446676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446494    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zjt\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-kube-api-access-52zjt\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.446676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446543    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.446676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446570    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.446676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446632    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57019be7-db39-44a3-8807-b65b7d577328-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.446676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.446663    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57019be7-db39-44a3-8807-b65b7d577328-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548376    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548419    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548440    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zjt\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-kube-api-access-52zjt\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548550    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548598    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548674    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57019be7-db39-44a3-8807-b65b7d577328-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548708    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57019be7-db39-44a3-8807-b65b7d577328-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548981 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548867    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548981 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548888    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548981 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548905    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.548981 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.548943    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.549172 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.549153    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.549244 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.549224    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.549752 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.549687    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57019be7-db39-44a3-8807-b65b7d577328-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.551216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.551194    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.551417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.551398    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57019be7-db39-44a3-8807-b65b7d577328-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.609213 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.609143    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.609543 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.609327    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zjt\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-kube-api-access-52zjt\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:55.655020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:55.654982    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:56.134701 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.134670    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"]
Nov 02 01:48:56.138223 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:48:56.138168    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57019be7_db39_44a3_8807_b65b7d577328.slice/crio-efbc9ce756c929e9f66ae0dee28dd383fb5108bddca1d5f0591b1808f897c525 WatchSource:0}: Error finding container efbc9ce756c929e9f66ae0dee28dd383fb5108bddca1d5f0591b1808f897c525: Status 404 returned error can't find the container with id efbc9ce756c929e9f66ae0dee28dd383fb5108bddca1d5f0591b1808f897c525
Nov 02 01:48:56.141575 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.141534    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:56.143123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.142522    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:56.143123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.142597    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:56.649139 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.649107    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"]
Nov 02 01:48:56.653318 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.653295    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.693287 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.693254    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" event={"ID":"57019be7-db39-44a3-8807-b65b7d577328","Type":"ContainerStarted","Data":"99e1809f9f7c62bf863762dc1e2fcdd411ab540845eb2e5cd9de86b3b89cdaa3"}
Nov 02 01:48:56.693489 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.693308    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" event={"ID":"57019be7-db39-44a3-8807-b65b7d577328","Type":"ContainerStarted","Data":"efbc9ce756c929e9f66ae0dee28dd383fb5108bddca1d5f0591b1808f897c525"}
Nov 02 01:48:56.710877 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.710847    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5fc56410-openshift-default-dockercfg-ppxm4\""
Nov 02 01:48:56.716951 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.716914    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"]
Nov 02 01:48:56.771833 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.771798    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-credential-socket\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.771833 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.771844    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-data\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.772057 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.771983    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xx8\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-kube-api-access-r4xx8\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.772095 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.772058    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-socket\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.772135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.772103    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-envoy\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.772246 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.772229    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-podinfo\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.772305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.772271    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-token\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.772383 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.772371    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-certs\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.772430 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.772423    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istiod-ca-cert\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.873920 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.873883    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-podinfo\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.874123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.873935    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-token\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.874123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.874006    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-certs\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.874123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.874050    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istiod-ca-cert\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.874123 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.874109    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-credential-socket\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.874351 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.874152    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-data\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.874351 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.874200    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xx8\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-kube-api-access-r4xx8\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.874351 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.874280    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-socket\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.874351 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.874333    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-envoy\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.875020 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.874934    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istiod-ca-cert\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.875213 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.875191    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-certs\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.875284 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.875211    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-data\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.875445 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.875414    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-credential-socket\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.875567 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.875488    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-socket\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.876950 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.876928    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-podinfo\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.877712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.877693    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-envoy\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.935301 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.935227    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xx8\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-kube-api-access-r4xx8\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.935301 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.935232    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-token\") pod \"e2e-ait-5fc56410-openshift-default-796d4cd445-5f765\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") " pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:56.970638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:56.969901    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:57.472633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.472124    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podStartSLOduration=2.47210274 podStartE2EDuration="2.47210274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:56.902053402 +0000 UTC m=+1444.775694452" watchObservedRunningTime="2026-11-02 01:48:57.47210274 +0000 UTC m=+1445.345743793"
Nov 02 01:48:57.482636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.481795    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"]
Nov 02 01:48:57.487213 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:48:57.487175    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddd4ab0_bc4f_4967_93e2_630bb55b3d02.slice/crio-51ddb16cc08650269e7f7c93c30983eb675340b4b052da9a601023c7ad5534e4 WatchSource:0}: Error finding container 51ddb16cc08650269e7f7c93c30983eb675340b4b052da9a601023c7ad5534e4: Status 404 returned error can't find the container with id 51ddb16cc08650269e7f7c93c30983eb675340b4b052da9a601023c7ad5534e4
Nov 02 01:48:57.489924 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.489894    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:57.490011 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.489973    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:57.490050 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.490011    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:48:57.660633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.657989    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:48:57.664631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.662746    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:48:57.664631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.662807    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 01:48:57.706637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.706014    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" event={"ID":"addd4ab0-bc4f-4967-93e2-630bb55b3d02","Type":"ContainerStarted","Data":"3b075bb28e989ce7072ef4bf1598ca3bbe6cbc843cc9a1039bb2603c78bf1d14"}
Nov 02 01:48:57.706637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.706064    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" event={"ID":"addd4ab0-bc4f-4967-93e2-630bb55b3d02","Type":"ContainerStarted","Data":"51ddb16cc08650269e7f7c93c30983eb675340b4b052da9a601023c7ad5534e4"}
Nov 02 01:48:57.829418 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.829355    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" podStartSLOduration=1.8293420820000001 podStartE2EDuration="1.829342082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:57.827841101 +0000 UTC m=+1445.701482154" watchObservedRunningTime="2026-11-02 01:48:57.829342082 +0000 UTC m=+1445.702983134"
Nov 02 01:48:57.974635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:57.973498    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:58.667633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:58.667035    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:48:58.667633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:58.667102    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 01:48:58.979859 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:58.979778    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:59.662636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:59.657018    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:48:59.662636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:59.657081    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 01:48:59.719875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:59.719845    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:48:59.720846 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:48:59.720827    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:49:00.658185 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:00.658148    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:49:00.658489 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:00.658460    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 01:49:01.663633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:01.663064    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:49:01.663633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:01.663151    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 01:49:02.655626 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:02.655571    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 01:49:02.655799 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:02.655660    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 01:49:03.758310 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:03.758273    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:49:03.759778 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:03.759752    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:49:03.760057 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:03.760040    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:49:07.015635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:06.925673    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 02 01:49:07.559317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:07.559273    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:07.574880 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:07.574831    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:07.950698 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:07.949699    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 02 01:49:07.989221 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:07.973724    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 02 01:49:07.989221 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:07.987103    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 02 01:49:08.031635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.020939    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.031635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.021628    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:49:08.380633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.379735    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.601796 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.601747    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.609636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.602364    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 01:49:08.686793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.686701    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:49:08.703794 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.703767    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:49:08.853638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.851104    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 01:49:08.971694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.971156    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:08.971881 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:08.971841    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:49:09.038754 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.038713    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:09.063734 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.063687    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.393819 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.393775    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.467208 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.467176    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:49:09.468357 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.426749    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.613633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.610877    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Nov 02 01:49:09.627961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.626702    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.788633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.769700    4287 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.891971 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.865661    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.932647 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.927812    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:09.932647 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.928231    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:49:09.932647 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.928275    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.013665 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:09.988740    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:49:10.237657 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:10.235180    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 01:49:10.624684 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:10.624652    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 01:49:10.628815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:10.628791    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 01:49:10.639088 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:10.631590    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 01:49:16.039830 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:49:16.039788    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:49:16.452455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.452376    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"]
Nov 02 01:49:16.456400 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.456380    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.534826 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.534716    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 01:49:16.535046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.534818    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 01:49:16.574235 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.574207    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 01:49:16.574415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.574241    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6ghtm\""
Nov 02 01:49:16.586641 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.586580    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.593242 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.593206    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.593559 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.593528    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8lb\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-kube-api-access-2k8lb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.593727 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.593621    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.593948 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.593930    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.594065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.594051    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.594218 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.594200    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.595740 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.595720    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"]
Nov 02 01:49:16.695370 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.695303    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.695564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.695431    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.695564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.695459    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.695564 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.695532    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.695783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.695555    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.695783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.695735    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8lb\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-kube-api-access-2k8lb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.695885 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.695789    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.696932 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.696903    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.698226 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.698199    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.698502 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.698414    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.698587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.698418    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.698693 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.698633    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.756348 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.756273    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.756486 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.756460    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8lb\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-kube-api-access-2k8lb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jmjhq\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:16.770413 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:16.770350    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:17.207773 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:17.207741    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"]
Nov 02 01:49:17.208463 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:49:17.208439    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfd0ee8_150f_44ad_bae5_52f7142ae3cc.slice/crio-e756d7b71fc31c479004acaf6fdb20029e220873166b4386db870d2ea489200d WatchSource:0}: Error finding container e756d7b71fc31c479004acaf6fdb20029e220873166b4386db870d2ea489200d: Status 404 returned error can't find the container with id e756d7b71fc31c479004acaf6fdb20029e220873166b4386db870d2ea489200d
Nov 02 01:49:18.125255 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:18.125216    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq" event={"ID":"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc","Type":"ContainerStarted","Data":"e756d7b71fc31c479004acaf6fdb20029e220873166b4386db870d2ea489200d"}
Nov 02 01:49:20.927537 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:20.927494    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:49:20.927837 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:20.927565    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:49:21.154047 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:21.154010    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq" event={"ID":"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc","Type":"ContainerStarted","Data":"1b58e31f20269aa2cd680f8f724dee3fa6d9f7797bf2e3b466b938eece6b788f"}
Nov 02 01:49:21.154253 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:21.154241    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:21.156180 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:21.156153    4287 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jmjhq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:49:21.156316 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:21.156213    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq" podUID="bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:49:21.361161 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:21.361092    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq" podStartSLOduration=1.645296261 podStartE2EDuration="5.361077831s" totalImagesPullingTime="3.71578157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:21.359447503 +0000 UTC m=+1469.233088555" watchObservedRunningTime="2026-11-02 01:49:21.361077831 +0000 UTC m=+1469.234718889"
Nov 02 01:49:22.166136 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:22.166096    4287 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jmjhq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:49:22.166597 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:22.166154    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq" podUID="bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:49:25.167816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.167776    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:49:25.395892 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.395854    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"]
Nov 02 01:49:25.400127 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.400101    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.471189 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.471100    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"]
Nov 02 01:49:25.518901 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.518852    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.519108 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.518942    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.519108 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.519061    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9364f0-0590-4807-84ff-a579394bf398-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.519245 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.519128    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.519245 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.519202    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.519369 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.519294    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsg82\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-kube-api-access-xsg82\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.519369 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.519348    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9364f0-0590-4807-84ff-a579394bf398-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.519480 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.519380    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.519480 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.519404    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.620561 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620521    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.620785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620583    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.620785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620664    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9364f0-0590-4807-84ff-a579394bf398-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.620785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620687    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.620785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620732    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621044 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620796    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsg82\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-kube-api-access-xsg82\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621044 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620840    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9364f0-0590-4807-84ff-a579394bf398-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621044 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620875    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621044 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.620900    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621044 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.621010    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621291 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.621118    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621347 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.621300    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621597 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.621445    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.621597 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.621449    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9364f0-0590-4807-84ff-a579394bf398-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.623338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.623304    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.623471 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.623452    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9364f0-0590-4807-84ff-a579394bf398-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.682437 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.682399    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.682586 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.682491    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsg82\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-kube-api-access-xsg82\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:25.714574 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:25.714537    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:26.147556 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:26.147511    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"]
Nov 02 01:49:26.151127 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:49:26.151092    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9364f0_0590_4807_84ff_a579394bf398.slice/crio-432ba7d907ce148f44b7793b15472bb228a16734620bac091a6f963d335253f9 WatchSource:0}: Error finding container 432ba7d907ce148f44b7793b15472bb228a16734620bac091a6f963d335253f9: Status 404 returned error can't find the container with id 432ba7d907ce148f44b7793b15472bb228a16734620bac091a6f963d335253f9
Nov 02 01:49:26.154237 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:26.154202    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:49:26.154335 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:26.154271    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:49:26.154335 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:26.154301    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:49:26.195815 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:26.195779    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" event={"ID":"ed9364f0-0590-4807-84ff-a579394bf398","Type":"ContainerStarted","Data":"432ba7d907ce148f44b7793b15472bb228a16734620bac091a6f963d335253f9"}
Nov 02 01:49:27.203567 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:27.203528    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" event={"ID":"ed9364f0-0590-4807-84ff-a579394bf398","Type":"ContainerStarted","Data":"fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1"}
Nov 02 01:49:27.426512 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:27.426455    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podStartSLOduration=2.426438897 podStartE2EDuration="2.426438897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:27.374215436 +0000 UTC m=+1475.247856510" watchObservedRunningTime="2026-11-02 01:49:27.426438897 +0000 UTC m=+1475.300079949"
Nov 02 01:49:27.715510 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:27.715471    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:27.717112 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:27.717085    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 01:49:27.717234 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:27.717136    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 01:49:28.715679 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:28.715648    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 01:49:28.716160 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:28.715705    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 01:49:29.715627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:29.715570    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 01:49:29.715821 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:29.715673    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 01:49:30.715518 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:30.715483    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 01:49:30.715696 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:30.715536    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 01:49:31.715355 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:31.715318    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 01:49:31.715864 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:31.715376    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 01:49:32.715145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:32.715104    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 01:49:32.715328 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:32.715173    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 01:49:33.719136 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:33.719107    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:33.719513 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:33.719472    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:33.720075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:33.720055    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 01:49:45.909898 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:45.909865    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"]
Nov 02 01:49:45.914175 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:45.914152    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.017051 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.017025    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2821b375-serving-cert\""
Nov 02 01:49:46.021017 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.020991    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"]
Nov 02 01:49:46.045076 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:49:46.045042    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:49:46.057938 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.057906    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-metrics-certs\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.058101 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.057982    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzkmx\" (UniqueName: \"kubernetes.io/projected/396dcebd-d0c6-43ea-a864-7ad8154787f6-kube-api-access-jzkmx\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.058202 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.058176    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396dcebd-d0c6-43ea-a864-7ad8154787f6-openshift-service-ca\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.058263 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.058248    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.160314 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.160276    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396dcebd-d0c6-43ea-a864-7ad8154787f6-openshift-service-ca\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.160497 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.160365    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.160559 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:49:46.160518    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-serving-cert: secret "maas-api-e2e-ait-2821b375-serving-cert" not found
Nov 02 01:49:46.160597 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.160558    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-metrics-certs\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.160597 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:49:46.160578    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls podName:396dcebd-d0c6-43ea-a864-7ad8154787f6 nodeName:}" failed. No retries permitted until 2026-11-02 01:49:46.660562936 +0000 UTC m=+1494.534203983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls") pod "maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7" (UID: "396dcebd-d0c6-43ea-a864-7ad8154787f6") : secret "maas-api-e2e-ait-2821b375-serving-cert" not found
Nov 02 01:49:46.160719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.160677    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkmx\" (UniqueName: \"kubernetes.io/projected/396dcebd-d0c6-43ea-a864-7ad8154787f6-kube-api-access-jzkmx\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.161208 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.161181    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396dcebd-d0c6-43ea-a864-7ad8154787f6-openshift-service-ca\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.165054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.165032    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-metrics-certs\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.223233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.223199    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzkmx\" (UniqueName: \"kubernetes.io/projected/396dcebd-d0c6-43ea-a864-7ad8154787f6-kube-api-access-jzkmx\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.668458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.668420    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.671127 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.671108    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls\") pod \"maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:46.828271 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:46.828244    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:47.214307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:47.214274    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"]
Nov 02 01:49:47.214745 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:49:47.214526    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396dcebd_d0c6_43ea_a864_7ad8154787f6.slice/crio-642fe09b7035e6700724dc69bdbf320e6578bfa334a1ae6b361c2a01ccb93b74 WatchSource:0}: Error finding container 642fe09b7035e6700724dc69bdbf320e6578bfa334a1ae6b361c2a01ccb93b74: Status 404 returned error can't find the container with id 642fe09b7035e6700724dc69bdbf320e6578bfa334a1ae6b361c2a01ccb93b74
Nov 02 01:49:47.367133 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:47.367094    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7" event={"ID":"396dcebd-d0c6-43ea-a864-7ad8154787f6","Type":"ContainerStarted","Data":"642fe09b7035e6700724dc69bdbf320e6578bfa334a1ae6b361c2a01ccb93b74"}
Nov 02 01:49:48.386316 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:48.386258    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7" event={"ID":"396dcebd-d0c6-43ea-a864-7ad8154787f6","Type":"ContainerStarted","Data":"2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8"}
Nov 02 01:49:48.386844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:48.386370    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:49:48.503759 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:48.503641    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7" podStartSLOduration=2.560768501 podStartE2EDuration="3.503623686s" totalImagesPullingTime="942.855185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:48.502767081 +0000 UTC m=+1496.376408134" watchObservedRunningTime="2026-11-02 01:49:48.503623686 +0000 UTC m=+1496.377264736"
Nov 02 01:49:52.429450 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:52.429422    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:49:52.461195 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:52.461167    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:49:54.404239 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:49:54.404211    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:50:15.241017 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.240976    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"]
Nov 02 01:50:15.245456 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.245431    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.323557 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.323522    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.323557 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.323568    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.323820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.323591    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.323820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.323701    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.323932 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.323886    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.324002 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.323983    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.324072 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.324056    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43fbecd-4574-4bcb-b700-7d9da3d08656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.324179 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.324165    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vzc\" (UniqueName: \"kubernetes.io/projected/e43fbecd-4574-4bcb-b700-7d9da3d08656-kube-api-access-x2vzc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.324251 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.324237    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.347632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.347584    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjf7g\""
Nov 02 01:50:15.359342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.359303    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"]
Nov 02 01:50:15.426059 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426026    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43fbecd-4574-4bcb-b700-7d9da3d08656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426126    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vzc\" (UniqueName: \"kubernetes.io/projected/e43fbecd-4574-4bcb-b700-7d9da3d08656-kube-api-access-x2vzc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426185    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426412 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426286    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426412 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426324    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426412 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426362    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426412 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426388    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426665 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426481    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426665 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426563    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426810 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426784    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426939 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426863    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.426939 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.426808    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.427189 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.427162    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.427290 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.427275    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e43fbecd-4574-4bcb-b700-7d9da3d08656-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.429515 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.429479    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.429675 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.429651    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.487478 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.487444    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e43fbecd-4574-4bcb-b700-7d9da3d08656-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.487694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.487658    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vzc\" (UniqueName: \"kubernetes.io/projected/e43fbecd-4574-4bcb-b700-7d9da3d08656-kube-api-access-x2vzc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-p5tbl\" (UID: \"e43fbecd-4574-4bcb-b700-7d9da3d08656\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:15.580124 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:15.580083    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:16.020214 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:50:16.020184    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43fbecd_4574_4bcb_b700_7d9da3d08656.slice/crio-ad317d3ed709ebc9c65089539b0480dab0aa649d15694a265e5c8c9b78f5b852 WatchSource:0}: Error finding container ad317d3ed709ebc9c65089539b0480dab0aa649d15694a265e5c8c9b78f5b852: Status 404 returned error can't find the container with id ad317d3ed709ebc9c65089539b0480dab0aa649d15694a265e5c8c9b78f5b852
Nov 02 01:50:16.020834 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:16.020809    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"]
Nov 02 01:50:16.023165 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:16.023135    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:16.023248 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:16.023201    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:16.023248 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:16.023230    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:16.047598 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:50:16.047561    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:50:16.693437 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:16.693394    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" event={"ID":"e43fbecd-4574-4bcb-b700-7d9da3d08656","Type":"ContainerStarted","Data":"c770304f87bd5b4ee3b5d7586cc62dae45ef54b7b97dad3f6236a7c7f39165c6"}
Nov 02 01:50:16.693870 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:16.693471    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" event={"ID":"e43fbecd-4574-4bcb-b700-7d9da3d08656","Type":"ContainerStarted","Data":"ad317d3ed709ebc9c65089539b0480dab0aa649d15694a265e5c8c9b78f5b852"}
Nov 02 01:50:16.817058 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:16.816994    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podStartSLOduration=1.81697359 podStartE2EDuration="1.81697359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:16.8138521 +0000 UTC m=+1524.687493155" watchObservedRunningTime="2026-11-02 01:50:16.81697359 +0000 UTC m=+1524.690614644"
Nov 02 01:50:17.585660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:17.584492    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:17.585660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:17.584646    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 01:50:17.585660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:17.584697    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 01:50:18.581335 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:18.581295    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 01:50:18.581796 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:18.581387    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 01:50:19.581170 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:19.581131    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 01:50:19.581349 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:19.581184    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 01:50:20.582811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:20.582307    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 01:50:20.582811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:20.582381    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 01:50:21.581054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:21.581010    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 01:50:21.581311 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:21.581072    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 01:50:21.732155 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:21.732125    4287 generic.go:356] "Generic (PLEG): container finished" podID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerID="f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9" exitCode=0
Nov 02 01:50:21.732552 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:21.732195    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" event={"ID":"25048cc7-4e48-4699-a20c-5740ae3065e0","Type":"ContainerDied","Data":"f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9"}
Nov 02 01:50:21.732655 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:21.732641    4287 scope.go:122] "RemoveContainer" containerID="f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9"
Nov 02 01:50:21.734288 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:21.734264    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:21.734370 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:21.734327    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:21.734411 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:21.734370    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:50:22.586633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:22.583042    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 01:50:22.586633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:22.583109    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 01:50:22.741097 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:22.741067    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" event={"ID":"25048cc7-4e48-4699-a20c-5740ae3065e0","Type":"ContainerStarted","Data":"87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103"}
Nov 02 01:50:23.571716 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:23.571679    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:50:23.572311 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:23.571983    4287 patch_prober.go:29] interesting pod/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:23.572311 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:23.572039    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 01:50:23.581340 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:23.581307    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 02 01:50:23.581476 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:23.581366    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 02 01:50:24.580094 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:24.580049    4287 patch_prober.go:29] interesting pod/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:24.580531 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:24.580122    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 01:50:24.677634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:24.675706    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:50:25.517789 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:25.517755    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:25.518320 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:25.518295    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:25.521223 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:25.521196    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl"
Nov 02 01:50:25.570707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:25.570668    4287 patch_prober.go:29] interesting pod/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:25.571232 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:25.570943    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 01:50:26.570415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:26.569972    4287 patch_prober.go:29] interesting pod/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:26.570415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:26.570034    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 01:50:27.570073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:27.570031    4287 patch_prober.go:29] interesting pod/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 02 01:50:27.570380 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:27.570346    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 02 01:50:28.574557 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:28.574520    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:50:28.575643 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:50:28.575600    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:50:46.040290 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:50:46.040248    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:51:00.762820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:00.762776    4287 patch_prober.go:29] interesting pod/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 01:51:00.763354 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:00.762844    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 01:51:01.076160 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:01.076076    4287 generic.go:356] "Generic (PLEG): container finished" podID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerID="3b075bb28e989ce7072ef4bf1598ca3bbe6cbc843cc9a1039bb2603c78bf1d14" exitCode=0
Nov 02 01:51:01.076333 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:01.076149    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" event={"ID":"addd4ab0-bc4f-4967-93e2-630bb55b3d02","Type":"ContainerDied","Data":"3b075bb28e989ce7072ef4bf1598ca3bbe6cbc843cc9a1039bb2603c78bf1d14"}
Nov 02 01:51:01.076661 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:01.076643    4287 scope.go:122] "RemoveContainer" containerID="3b075bb28e989ce7072ef4bf1598ca3bbe6cbc843cc9a1039bb2603c78bf1d14"
Nov 02 01:51:01.078302 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:01.078275    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:51:01.078387 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:01.078368    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:51:01.078427 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:01.078402    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:51:02.087135 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:02.087076    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" event={"ID":"addd4ab0-bc4f-4967-93e2-630bb55b3d02","Type":"ContainerStarted","Data":"6fd4fdab317687bea41af9aa5788d27cfb7b3050f134ab0d385042822b03e4a6"}
Nov 02 01:51:02.977649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:02.977047    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:51:02.977649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:02.977182    4287 patch_prober.go:29] interesting pod/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:51:02.977649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:02.977227    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 01:51:03.977631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:03.974008    4287 patch_prober.go:29] interesting pod/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:51:03.977631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:03.974071    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 01:51:04.974075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:04.973980    4287 patch_prober.go:29] interesting pod/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:51:04.974075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:04.974042    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 01:51:05.974081 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:05.974037    4287 patch_prober.go:29] interesting pod/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:51:05.974497 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:05.974106    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 01:51:06.974008 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:06.973964    4287 patch_prober.go:29] interesting pod/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 02 01:51:06.974214 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:06.974041    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 02 01:51:07.977026 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:07.976990    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:51:07.977489 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:07.977369    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:51:07.978288 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:07.978270    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:51:16.039982 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:51:16.039941    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:51:44.564713 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:44.564670    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"]
Nov 02 01:51:44.565299 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:44.564932    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7" podUID="396dcebd-d0c6-43ea-a864-7ad8154787f6" containerName="maas-api" containerID="cri-o://2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8" gracePeriod=30
Nov 02 01:51:44.900379 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:44.900348    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:51:45.026030 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.026001    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396dcebd-d0c6-43ea-a864-7ad8154787f6-kube-api-access-jzkmx\" (UniqueName: \"kubernetes.io/projected/396dcebd-d0c6-43ea-a864-7ad8154787f6-kube-api-access-jzkmx\") pod \"396dcebd-d0c6-43ea-a864-7ad8154787f6\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") "
Nov 02 01:51:45.026224 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.026061    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396dcebd-d0c6-43ea-a864-7ad8154787f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396dcebd-d0c6-43ea-a864-7ad8154787f6-openshift-service-ca\") pod \"396dcebd-d0c6-43ea-a864-7ad8154787f6\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") "
Nov 02 01:51:45.026224 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.026106    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-metrics-certs\") pod \"396dcebd-d0c6-43ea-a864-7ad8154787f6\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") "
Nov 02 01:51:45.026224 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.026152    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls\") pod \"396dcebd-d0c6-43ea-a864-7ad8154787f6\" (UID: \"396dcebd-d0c6-43ea-a864-7ad8154787f6\") "
Nov 02 01:51:45.026499 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.026458    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396dcebd-d0c6-43ea-a864-7ad8154787f6-openshift-service-ca" pod "396dcebd-d0c6-43ea-a864-7ad8154787f6" (UID: "396dcebd-d0c6-43ea-a864-7ad8154787f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:45.026973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.026947    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/396dcebd-d0c6-43ea-a864-7ad8154787f6-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:45.028853 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.028827    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396dcebd-d0c6-43ea-a864-7ad8154787f6-kube-api-access-jzkmx" pod "396dcebd-d0c6-43ea-a864-7ad8154787f6" (UID: "396dcebd-d0c6-43ea-a864-7ad8154787f6"). InnerVolumeSpecName "kube-api-access-jzkmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:45.028950 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.028885    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-metrics-certs" pod "396dcebd-d0c6-43ea-a864-7ad8154787f6" (UID: "396dcebd-d0c6-43ea-a864-7ad8154787f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:45.029060 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.029038    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls" pod "396dcebd-d0c6-43ea-a864-7ad8154787f6" (UID: "396dcebd-d0c6-43ea-a864-7ad8154787f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:51:45.128940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.128791    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzkmx\" (UniqueName: \"kubernetes.io/projected/396dcebd-d0c6-43ea-a864-7ad8154787f6-kube-api-access-jzkmx\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:45.128940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.128832    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:45.128940 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.128852    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/396dcebd-d0c6-43ea-a864-7ad8154787f6-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:45.457834 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.457753    4287 generic.go:356] "Generic (PLEG): container finished" podID="396dcebd-d0c6-43ea-a864-7ad8154787f6" containerID="2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8" exitCode=0
Nov 02 01:51:45.457834 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.457801    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"
Nov 02 01:51:45.457834 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.457812    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7" event={"ID":"396dcebd-d0c6-43ea-a864-7ad8154787f6","Type":"ContainerDied","Data":"2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8"}
Nov 02 01:51:45.458073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.457862    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7" event={"ID":"396dcebd-d0c6-43ea-a864-7ad8154787f6","Type":"ContainerDied","Data":"642fe09b7035e6700724dc69bdbf320e6578bfa334a1ae6b361c2a01ccb93b74"}
Nov 02 01:51:45.458073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.457892    4287 scope.go:122] "RemoveContainer" containerID="2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8"
Nov 02 01:51:45.469470 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.469447    4287 scope.go:122] "RemoveContainer" containerID="2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8"
Nov 02 01:51:45.469818 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:51:45.469795    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8\": container with ID starting with 2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8 not found: ID does not exist" containerID="2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8"
Nov 02 01:51:45.469891 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.469834    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8"} err="failed to get container status \"2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8\": rpc error: code = NotFound desc = could not find container \"2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8\": container with ID starting with 2445967ee71b9259763178287027ab957dc735b586c6157da5b09ed830c83ca8 not found: ID does not exist"
Nov 02 01:51:45.674034 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.673996    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"]
Nov 02 01:51:45.701587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.701556    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2821b375-84c58dc75c-fjdg7"]
Nov 02 01:51:45.941303 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.941270    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"]
Nov 02 01:51:45.941939 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.941923    4287 cpu_manager.go:414] "containerMap: removing container" podUID="396dcebd-d0c6-43ea-a864-7ad8154787f6" containerName="maas-api"
Nov 02 01:51:45.942010 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.941941    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396dcebd-d0c6-43ea-a864-7ad8154787f6" containerName="maas-api"
Nov 02 01:51:45.942059 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.942049    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="396dcebd-d0c6-43ea-a864-7ad8154787f6" containerName="maas-api"
Nov 02 01:51:45.947144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.947119    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.996652 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.996548    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.996652 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.996589    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3e385c-906e-4f68-855a-f299826eb33b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.996926 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.996651    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.996926 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.996677    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkdt8\" (UniqueName: \"kubernetes.io/projected/8f3e385c-906e-4f68-855a-f299826eb33b-kube-api-access-nkdt8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.996926 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.996781    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.997089 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.996981    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3e385c-906e-4f68-855a-f299826eb33b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.997262 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.997235    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3e385c-906e-4f68-855a-f299826eb33b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.997375 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.997318    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:45.997375 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:45.997348    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.017902 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.017871    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"]
Nov 02 01:51:46.057862 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:51:46.057821    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:51:46.098830 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.098790    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3e385c-906e-4f68-855a-f299826eb33b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099055 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.098911    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3e385c-906e-4f68-855a-f299826eb33b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099055 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.098975    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099055 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.099007    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099055 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.099032    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099395 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.099056    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3e385c-906e-4f68-855a-f299826eb33b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099395 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.099084    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099395 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.099109    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkdt8\" (UniqueName: \"kubernetes.io/projected/8f3e385c-906e-4f68-855a-f299826eb33b-kube-api-access-nkdt8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099395 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.099203    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.099758 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.099733    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.100392 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.100370    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.101348 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.100800    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.101483 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.101136    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3e385c-906e-4f68-855a-f299826eb33b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.101579 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.101295    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.102944 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.102900    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3e385c-906e-4f68-855a-f299826eb33b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.104394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.104354    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3e385c-906e-4f68-855a-f299826eb33b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.194216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.194145    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3e385c-906e-4f68-855a-f299826eb33b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.237011 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.233324    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkdt8\" (UniqueName: \"kubernetes.io/projected/8f3e385c-906e-4f68-855a-f299826eb33b-kube-api-access-nkdt8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8x8rj\" (UID: \"8f3e385c-906e-4f68-855a-f299826eb33b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.249116 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.249015    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396dcebd-d0c6-43ea-a864-7ad8154787f6" path="/var/lib/kubelet/pods/396dcebd-d0c6-43ea-a864-7ad8154787f6/volumes"
Nov 02 01:51:46.265309 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.265260    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:46.770283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.770250    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"]
Nov 02 01:51:46.770669 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:51:46.770461    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3e385c_906e_4f68_855a_f299826eb33b.slice/crio-38704f8be0c9946d3e21f79524662d3bb81023733c13db1026dff1c8e26b3e9f WatchSource:0}: Error finding container 38704f8be0c9946d3e21f79524662d3bb81023733c13db1026dff1c8e26b3e9f: Status 404 returned error can't find the container with id 38704f8be0c9946d3e21f79524662d3bb81023733c13db1026dff1c8e26b3e9f
Nov 02 01:51:46.772973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.772940    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:51:46.773077 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.773004    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:51:46.773077 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:46.773034    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:51:47.482294 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:47.482256    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" event={"ID":"8f3e385c-906e-4f68-855a-f299826eb33b","Type":"ContainerStarted","Data":"acaeee8e13cea99f5251a54e3e6dd3c2260efe12a5afb5becf76fe4c8d2f0b69"}
Nov 02 01:51:47.482481 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:47.482323    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" event={"ID":"8f3e385c-906e-4f68-855a-f299826eb33b","Type":"ContainerStarted","Data":"38704f8be0c9946d3e21f79524662d3bb81023733c13db1026dff1c8e26b3e9f"}
Nov 02 01:51:47.772894 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:47.723327    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podStartSLOduration=2.7233082299999998 podStartE2EDuration="2.72330823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:47.666535192 +0000 UTC m=+1615.540176245" watchObservedRunningTime="2026-11-02 01:51:47.72330823 +0000 UTC m=+1615.596949282"
Nov 02 01:51:48.270124 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:48.269731    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:51:48.270124 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:48.269790    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 01:51:48.274734 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:48.272409    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:48.712122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:48.710507    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"]
Nov 02 01:51:48.712122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:48.710810    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy" containerID="cri-o://87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103" gracePeriod=30
Nov 02 01:51:49.270637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:49.266997    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:51:49.270637 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:49.267060    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 01:51:50.274635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:50.274511    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:51:50.274635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:50.274574    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 01:51:51.270633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:51.268576    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:51:51.270633 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:51.268659    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 01:51:52.273973 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:52.273931    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:51:52.274364 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:52.274033    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 01:51:53.271064 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:53.271026    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:51:53.271368 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:53.271332    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 01:51:54.047268 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.047244    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:51:54.133443 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133410    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-certs\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.133657 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133507    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-envoy\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.133657 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133557    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-podinfo\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.133793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133672    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25048cc7-4e48-4699-a20c-5740ae3065e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25048cc7-4e48-4699-a20c-5740ae3065e0-istiod-ca-cert\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.133793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133754    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-certs" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.133899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133814    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-data\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.133899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133852    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-socket\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.134006 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133906    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-token\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-token\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.134006 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.133987    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-credential-socket\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.134109 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134019    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-kube-api-access-m886z\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-kube-api-access-m886z\") pod \"25048cc7-4e48-4699-a20c-5740ae3065e0\" (UID: \"25048cc7-4e48-4699-a20c-5740ae3065e0\") "
Nov 02 01:51:54.134231 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134182    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-socket" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.134231 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134119    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25048cc7-4e48-4699-a20c-5740ae3065e0-istiod-ca-cert" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:51:54.134529 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134359    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-data" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.134529 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134367    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-credential-socket" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.134783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134751    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.134783 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134778    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.134933 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134795    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.134933 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134809    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.134933 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.134822    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25048cc7-4e48-4699-a20c-5740ae3065e0-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.136279 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.136212    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-envoy" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:51:54.136978 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.136901    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-token" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:54.136978 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.136912    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-podinfo" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:51:54.136978 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.136963    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-kube-api-access-m886z" pod "25048cc7-4e48-4699-a20c-5740ae3065e0" (UID: "25048cc7-4e48-4699-a20c-5740ae3065e0"). InnerVolumeSpecName "kube-api-access-m886z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:51:54.235712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.235678    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.235712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.235712    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m886z\" (UniqueName: \"kubernetes.io/projected/25048cc7-4e48-4699-a20c-5740ae3065e0-kube-api-access-m886z\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.235712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.235728    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.235712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.235740    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25048cc7-4e48-4699-a20c-5740ae3065e0-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:51:54.266115 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.266040    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 01:51:54.266115 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.266093    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 01:51:54.558694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.558590    4287 generic.go:356] "Generic (PLEG): container finished" podID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerID="87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103" exitCode=0
Nov 02 01:51:54.558694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.558658    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"
Nov 02 01:51:54.558694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.558670    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" event={"ID":"25048cc7-4e48-4699-a20c-5740ae3065e0","Type":"ContainerDied","Data":"87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103"}
Nov 02 01:51:54.558947 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.558733    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb" event={"ID":"25048cc7-4e48-4699-a20c-5740ae3065e0","Type":"ContainerDied","Data":"7b43914da57a2bab1323112c168dc9a6a4bf17ef26bf0970f64adaddeee5e0ac"}
Nov 02 01:51:54.558947 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.558766    4287 scope.go:122] "RemoveContainer" containerID="87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103"
Nov 02 01:51:54.570587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.570564    4287 scope.go:122] "RemoveContainer" containerID="f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9"
Nov 02 01:51:54.582763 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.582722    4287 scope.go:122] "RemoveContainer" containerID="87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103"
Nov 02 01:51:54.583110 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:51:54.583078    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103\": container with ID starting with 87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103 not found: ID does not exist" containerID="87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103"
Nov 02 01:51:54.583240 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.583120    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103"} err="failed to get container status \"87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103\": rpc error: code = NotFound desc = could not find container \"87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103\": container with ID starting with 87cc48f4215f2b11fd24fa1f7c9ae993a637949b57e3a8bbeee6ec1b05ba2103 not found: ID does not exist"
Nov 02 01:51:54.583240 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.583142    4287 scope.go:122] "RemoveContainer" containerID="f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9"
Nov 02 01:51:54.583482 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:51:54.583461    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9\": container with ID starting with f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9 not found: ID does not exist" containerID="f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9"
Nov 02 01:51:54.583551 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.583493    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9"} err="failed to get container status \"f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9\": rpc error: code = NotFound desc = could not find container \"f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9\": container with ID starting with f68bbc06b467228781f06308b39fc2562deed5c85c12d7ba21ef5319aa4dfde9 not found: ID does not exist"
Nov 02 01:51:54.726777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.726742    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"]
Nov 02 01:51:54.732667 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:54.732642    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fd723454-openshift-default-57c6cf448f-4cmzb"]
Nov 02 01:51:56.134933 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:56.134902    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:56.135441 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:56.135416    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:56.136266 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:56.136246    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj"
Nov 02 01:51:56.242398 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:56.242368    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" path="/var/lib/kubelet/pods/25048cc7-4e48-4699-a20c-5740ae3065e0/volumes"
Nov 02 01:51:57.422126 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:57.422083    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"]
Nov 02 01:51:57.422654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:51:57.422310    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy" containerID="cri-o://6fd4fdab317687bea41af9aa5788d27cfb7b3050f134ab0d385042822b03e4a6" gracePeriod=30
Nov 02 01:52:01.079672 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.079635    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"]
Nov 02 01:52:01.081429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.080941    4287 cpu_manager.go:414] "containerMap: removing container" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy"
Nov 02 01:52:01.081429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.080965    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy"
Nov 02 01:52:01.081429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.081038    4287 cpu_manager.go:414] "containerMap: removing container" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy"
Nov 02 01:52:01.081429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.081046    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy"
Nov 02 01:52:01.081791 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.081479    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy"
Nov 02 01:52:01.081791 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.081496    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="25048cc7-4e48-4699-a20c-5740ae3065e0" containerName="istio-proxy"
Nov 02 01:52:01.088134 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.088100    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.202901 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.202572    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.202901 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.202815    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.203110 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.202938    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.203110 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.203038    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqxq5\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-kube-api-access-bqxq5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.203110 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.203077    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.203218 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.203202    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.203262 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.203248    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.203360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.203342    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.203401 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.203390    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.304839 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.304793    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"]
Nov 02 01:52:01.305470 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.305443    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.306024 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.306001    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.307270 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.307246    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.307390 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.307307    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.307458 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.307405    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.307583 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.307555    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.307773 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.307756    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.307846 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.307802    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqxq5\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-kube-api-access-bqxq5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.307846 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.307826    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.308181 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.308166    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.308249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.307189    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.308706 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.308681    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.309124 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.309095    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.309376 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.309352    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.311519 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.311493    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.312075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.312050    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.367111 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.367053    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.372363 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.372336    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqxq5\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-kube-api-access-bqxq5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xspvr\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.404795 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.404775    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:01.917043 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.917009    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"]
Nov 02 01:52:01.917433 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:52:01.917408    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4390e3_7a12_48cf_ae27_01e87d95d0d6.slice/crio-5a5057a52674524de803ce0b6595ebe968afa3a09f40e6b37e3c54d4c72a99ce WatchSource:0}: Error finding container 5a5057a52674524de803ce0b6595ebe968afa3a09f40e6b37e3c54d4c72a99ce: Status 404 returned error can't find the container with id 5a5057a52674524de803ce0b6595ebe968afa3a09f40e6b37e3c54d4c72a99ce
Nov 02 01:52:01.919996 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.919962    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:52:01.920075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.920027    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:52:01.920075 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:01.920055    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:52:02.640054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.637413    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" event={"ID":"2a4390e3-7a12-48cf-ae27-01e87d95d0d6","Type":"ContainerStarted","Data":"9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2"}
Nov 02 01:52:02.640054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.637463    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" event={"ID":"2a4390e3-7a12-48cf-ae27-01e87d95d0d6","Type":"ContainerStarted","Data":"5a5057a52674524de803ce0b6595ebe968afa3a09f40e6b37e3c54d4c72a99ce"}
Nov 02 01:52:02.640054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.640028    4287 generic.go:356] "Generic (PLEG): container finished" podID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerID="6fd4fdab317687bea41af9aa5788d27cfb7b3050f134ab0d385042822b03e4a6" exitCode=0
Nov 02 01:52:02.640523 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.640071    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" event={"ID":"addd4ab0-bc4f-4967-93e2-630bb55b3d02","Type":"ContainerDied","Data":"6fd4fdab317687bea41af9aa5788d27cfb7b3050f134ab0d385042822b03e4a6"}
Nov 02 01:52:02.640523 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.640114    4287 scope.go:122] "RemoveContainer" containerID="3b075bb28e989ce7072ef4bf1598ca3bbe6cbc843cc9a1039bb2603c78bf1d14"
Nov 02 01:52:02.772632 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.767841    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:52:02.791769 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.791709    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podStartSLOduration=1.791694289 podStartE2EDuration="1.791694289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:02.787918077 +0000 UTC m=+1630.661559129" watchObservedRunningTime="2026-11-02 01:52:02.791694289 +0000 UTC m=+1630.665335340"
Nov 02 01:52:02.830595 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.830566    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-podinfo\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.830762 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.830688    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-certs\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.830762 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.830712    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-credential-socket\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.830851 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.830837    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-kube-api-access-r4xx8\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-kube-api-access-r4xx8\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.830938 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.830925    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-token\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-token\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.831028 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.830986    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-certs" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:02.831028 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.830994    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istiod-ca-cert\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.831149 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.831033    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-credential-socket" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:02.831149 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.831089    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-data\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.831149 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.831142    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-socket\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.831305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.831175    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-envoy\") pod \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\" (UID: \"addd4ab0-bc4f-4967-93e2-630bb55b3d02\") "
Nov 02 01:52:02.831305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.831267    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istiod-ca-cert" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:52:02.831395 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.831302    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-data" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:02.831395 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.831375    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-socket" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:02.832089 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.832052    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:02.832089 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.832081    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:02.832305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.832097    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:02.832305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.832111    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:02.832305 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.832124    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:02.833085 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.833058    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-podinfo" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:52:02.833350 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.833333    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-token" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:02.833510 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.833492    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-kube-api-access-r4xx8" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "kube-api-access-r4xx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:02.833787 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.833765    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-envoy" pod "addd4ab0-bc4f-4967-93e2-630bb55b3d02" (UID: "addd4ab0-bc4f-4967-93e2-630bb55b3d02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:02.933881 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.933805    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4xx8\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-kube-api-access-r4xx8\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:02.933881 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.933837    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:02.933881 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.933847    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:02.933881 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:02.933855    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/addd4ab0-bc4f-4967-93e2-630bb55b3d02-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:03.405286 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:03.405250    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:03.406555 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:03.406528    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xspvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:52:03.406676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:03.406573    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 01:52:03.647782 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:03.647754    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"
Nov 02 01:52:03.648182 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:03.647794    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765" event={"ID":"addd4ab0-bc4f-4967-93e2-630bb55b3d02","Type":"ContainerDied","Data":"51ddb16cc08650269e7f7c93c30983eb675340b4b052da9a601023c7ad5534e4"}
Nov 02 01:52:03.648244 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:03.647852    4287 scope.go:122] "RemoveContainer" containerID="6fd4fdab317687bea41af9aa5788d27cfb7b3050f134ab0d385042822b03e4a6"
Nov 02 01:52:03.882716 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:03.882678    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"]
Nov 02 01:52:03.929666 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:03.929634    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5fc56410-openshift-default-796d4cd445-5f765"]
Nov 02 01:52:04.240844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:04.240811    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" path="/var/lib/kubelet/pods/addd4ab0-bc4f-4967-93e2-630bb55b3d02/volumes"
Nov 02 01:52:04.405346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:04.405303    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xspvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:52:04.405553 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:04.405371    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 01:52:05.405723 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:05.405686    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xspvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:52:05.406094 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:05.405740    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 01:52:06.405234 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:06.405198    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xspvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:52:06.405447 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:06.405249    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 01:52:07.406120 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:07.406084    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xspvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:52:07.406525 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:07.406141    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 01:52:08.406151 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:08.406111    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xspvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:52:08.406538 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:08.406167    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 01:52:09.405670 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:09.405639    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xspvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 01:52:09.405875 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:09.405698    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 01:52:10.409201 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:10.409165    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:10.409653 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:10.409552    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:10.410144 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:10.410126    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 01:52:16.040354 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:52:16.040317    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:52:43.525350 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:43.525264    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"]
Nov 02 01:52:43.525915 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:43.525504    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy" containerID="cri-o://d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c" gracePeriod=30
Nov 02 01:52:43.575482 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:43.575453    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"]
Nov 02 01:52:43.575718 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:43.575680    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy" containerID="cri-o://3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e" gracePeriod=30
Nov 02 01:52:46.039740 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:52:46.039692    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:52:47.652358 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:47.652315    4287 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:52:47.652851 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:47.652379    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:52:48.870980 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.870953    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:52:48.893194 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.893170    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:52:48.991338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991298    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-socket\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.991585 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991386    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-certs\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.991585 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991449    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-envoy\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.991585 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991499    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-socket\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.991585 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991544    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-podinfo\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.991585 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991579    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2403ac43-089f-4aae-827e-5a59b8bf5a51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2403ac43-089f-4aae-827e-5a59b8bf5a51-istiod-ca-cert\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.991844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991602    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-socket" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.991844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991717    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-envoy\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.991844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991731    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-certs" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.991844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991781    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-data\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.991844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991838    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-token\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-token\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.992106 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991906    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-token\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-token\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.992106 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991948    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istiod-ca-cert\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.992106 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.991965    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2403ac43-089f-4aae-827e-5a59b8bf5a51-istiod-ca-cert" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:52:48.992106 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992026    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-credential-socket\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.992106 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992059    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-data" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.992106 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992064    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-data\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992143    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-credential-socket\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992178    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-kube-api-access-zhn5r\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-kube-api-access-zhn5r\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992252    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-data" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992262    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-kube-api-access-lx5jc\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-kube-api-access-lx5jc\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992311    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-certs\") pod \"2403ac43-089f-4aae-827e-5a59b8bf5a51\" (UID: \"2403ac43-089f-4aae-827e-5a59b8bf5a51\") "
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992353    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-podinfo\") pod \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\" (UID: \"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd\") "
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992363    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-credential-socket" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992368    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istiod-ca-cert" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:52:48.992417 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.992405    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-socket" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.993104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993036    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.993104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993057    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.993104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993070    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.993104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993086    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.993104 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993101    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.993459 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993114    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.993459 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993128    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2403ac43-089f-4aae-827e-5a59b8bf5a51-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.993459 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993141    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:48.993717 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993490    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-credential-socket" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.993717 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.993623    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-certs" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.995424 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.995375    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-envoy" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.995424 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.995399    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-kube-api-access-zhn5r" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "kube-api-access-zhn5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:48.997174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.997043    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-token" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:48.997174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.997115    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-podinfo" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:52:48.997174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.997140    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-envoy" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:52:48.997174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.997144    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-token" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:48.997174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.997141    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-kube-api-access-lx5jc" pod "2403ac43-089f-4aae-827e-5a59b8bf5a51" (UID: "2403ac43-089f-4aae-827e-5a59b8bf5a51"). InnerVolumeSpecName "kube-api-access-lx5jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:52:48.997665 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:48.997642    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-podinfo" pod "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" (UID: "4f19908d-04f8-48a2-ab46-d3f6e9ff53fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:52:49.040421 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.040383    4287 generic.go:356] "Generic (PLEG): container finished" podID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerID="d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c" exitCode=0
Nov 02 01:52:49.040649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.040441    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"
Nov 02 01:52:49.040649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.040488    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" event={"ID":"2403ac43-089f-4aae-827e-5a59b8bf5a51","Type":"ContainerDied","Data":"d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c"}
Nov 02 01:52:49.040649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.040558    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh" event={"ID":"2403ac43-089f-4aae-827e-5a59b8bf5a51","Type":"ContainerDied","Data":"5cb51cc62b2baa793109862d59a357aedc500f9de32dd81659f4fd52969caffe"}
Nov 02 01:52:49.040649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.040586    4287 scope.go:122] "RemoveContainer" containerID="d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c"
Nov 02 01:52:49.049239 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.049204    4287 generic.go:356] "Generic (PLEG): container finished" podID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerID="3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e" exitCode=0
Nov 02 01:52:49.049360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.049241    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" event={"ID":"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd","Type":"ContainerDied","Data":"3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e"}
Nov 02 01:52:49.049360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.049275    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"
Nov 02 01:52:49.049360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.049300    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx" event={"ID":"4f19908d-04f8-48a2-ab46-d3f6e9ff53fd","Type":"ContainerDied","Data":"6ef47a111454844f48cd414d26258117b8245914bdeeada89c059e418d798a89"}
Nov 02 01:52:49.064389 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.064364    4287 scope.go:122] "RemoveContainer" containerID="d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c"
Nov 02 01:52:49.064800 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:52:49.064771    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c\": container with ID starting with d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c not found: ID does not exist" containerID="d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c"
Nov 02 01:52:49.064906 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.064813    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c"} err="failed to get container status \"d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c\": rpc error: code = NotFound desc = could not find container \"d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c\": container with ID starting with d821037e5271a3d183406742a569656e310ab2063204d4200976dcde9331f74c not found: ID does not exist"
Nov 02 01:52:49.064906 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.064838    4287 scope.go:122] "RemoveContainer" containerID="3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e"
Nov 02 01:52:49.077660 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.077599    4287 scope.go:122] "RemoveContainer" containerID="3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e"
Nov 02 01:52:49.078001 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:52:49.077965    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e\": container with ID starting with 3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e not found: ID does not exist" containerID="3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e"
Nov 02 01:52:49.078091 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.078002    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e"} err="failed to get container status \"3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e\": rpc error: code = NotFound desc = could not find container \"3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e\": container with ID starting with 3795c8ed8b21306b22038006cd2bd21374fac34baf8569b809ca0664ad88fd8e not found: ID does not exist"
Nov 02 01:52:49.094479 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094440    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094479 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094476    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhn5r\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-kube-api-access-zhn5r\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094495    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx5jc\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-kube-api-access-lx5jc\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094509    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094523    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094535    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094549    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094563    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094576    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.094725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.094590    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2403ac43-089f-4aae-827e-5a59b8bf5a51-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:52:49.338416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.338330    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"]
Nov 02 01:52:49.342107 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.342082    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-5wggx"]
Nov 02 01:52:49.451736 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.451680    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"]
Nov 02 01:52:49.455892 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:49.455865    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-znpzh"]
Nov 02 01:52:50.239770 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:50.239733    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" path="/var/lib/kubelet/pods/2403ac43-089f-4aae-827e-5a59b8bf5a51/volumes"
Nov 02 01:52:50.240490 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:52:50.240456    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" path="/var/lib/kubelet/pods/4f19908d-04f8-48a2-ab46-d3f6e9ff53fd/volumes"
Nov 02 01:53:16.040332 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:53:16.040292    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:53:29.475801 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.475765    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"]
Nov 02 01:53:29.476589 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476567    4287 cpu_manager.go:414] "containerMap: removing container" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy"
Nov 02 01:53:29.476738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476591    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy"
Nov 02 01:53:29.476738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476627    4287 cpu_manager.go:414] "containerMap: removing container" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy"
Nov 02 01:53:29.476738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476636    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy"
Nov 02 01:53:29.476738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476658    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy"
Nov 02 01:53:29.476738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476666    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy"
Nov 02 01:53:29.476983 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476836    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f19908d-04f8-48a2-ab46-d3f6e9ff53fd" containerName="istio-proxy"
Nov 02 01:53:29.476983 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476851    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy"
Nov 02 01:53:29.476983 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476861    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy"
Nov 02 01:53:29.476983 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.476869    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2403ac43-089f-4aae-827e-5a59b8bf5a51" containerName="istio-proxy"
Nov 02 01:53:29.480429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.480407    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.584196 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.584161    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-59842dd7-serving-cert\""
Nov 02 01:53:29.585356 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.585332    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-openshift-service-ca\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.585510 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.585377    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmm5l\" (UniqueName: \"kubernetes.io/projected/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-kube-api-access-tmm5l\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.585510 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.585405    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-metrics-certs\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.585510 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.585491    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.588353 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.588328    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"]
Nov 02 01:53:29.687064 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.687024    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-openshift-service-ca\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.687264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.687078    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmm5l\" (UniqueName: \"kubernetes.io/projected/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-kube-api-access-tmm5l\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.687264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.687105    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-metrics-certs\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.687264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.687132    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.687440 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:53:29.687316    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-serving-cert: secret "maas-api-e2e-mt-59842dd7-serving-cert" not found
Nov 02 01:53:29.687440 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:53:29.687386    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls podName:8ccfadf3-2ab6-4783-ab01-ebfdff800afb nodeName:}" failed. No retries permitted until 2026-11-02 01:53:30.187366565 +0000 UTC m=+1718.061007614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls") pod "maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47" (UID: "8ccfadf3-2ab6-4783-ab01-ebfdff800afb") : secret "maas-api-e2e-mt-59842dd7-serving-cert" not found
Nov 02 01:53:29.687753 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.687731    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-openshift-service-ca\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.689725 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.689704    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-metrics-certs\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:29.750791 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:29.750713    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmm5l\" (UniqueName: \"kubernetes.io/projected/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-kube-api-access-tmm5l\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:30.195508 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:30.195469    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:30.198421 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:30.198384    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls\") pod \"maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:30.394791 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:30.394761    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:30.780054 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:53:30.780010    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccfadf3_2ab6_4783_ab01_ebfdff800afb.slice/crio-c6925f31027d944b91f18556a2b979f2d48a51d613882db8573ad5ab1c647588 WatchSource:0}: Error finding container c6925f31027d944b91f18556a2b979f2d48a51d613882db8573ad5ab1c647588: Status 404 returned error can't find the container with id c6925f31027d944b91f18556a2b979f2d48a51d613882db8573ad5ab1c647588
Nov 02 01:53:30.780485 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:30.780112    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"]
Nov 02 01:53:31.328803 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:31.328765    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47" event={"ID":"8ccfadf3-2ab6-4783-ab01-ebfdff800afb","Type":"ContainerStarted","Data":"c6925f31027d944b91f18556a2b979f2d48a51d613882db8573ad5ab1c647588"}
Nov 02 01:53:32.337622 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:32.337511    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47" event={"ID":"8ccfadf3-2ab6-4783-ab01-ebfdff800afb","Type":"ContainerStarted","Data":"b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736"}
Nov 02 01:53:32.337622 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:32.337580    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:32.456935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:32.456880    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47" podStartSLOduration=2.198306015 podStartE2EDuration="3.456864704s" totalImagesPullingTime="1.258558689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:32.454061665 +0000 UTC m=+1720.327702715" watchObservedRunningTime="2026-11-02 01:53:32.456864704 +0000 UTC m=+1720.330505765"
Nov 02 01:53:38.349986 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:38.349957    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:53:46.040982 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:53:46.040939    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:53:46.529654 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.529601    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"]
Nov 02 01:53:46.530262 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.530247    4287 cpu_manager.go:414] "containerMap: removing container" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy"
Nov 02 01:53:46.530317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.530265    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addd4ab0-bc4f-4967-93e2-630bb55b3d02" containerName="istio-proxy"
Nov 02 01:53:46.533806 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.533789    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.558897 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.558854    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-socket\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.559261 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.559236    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-data\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.559480 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.559458    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfgw\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-kube-api-access-rbfgw\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.559689 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.559670    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-token\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.559870 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.559854    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-credential-socket\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.560030 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.560014    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-podinfo\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.560334 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.560311    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-envoy\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.560490 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.560456    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istiod-ca-cert\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.560665 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.560556    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-certs\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.613073 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.612994    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"]
Nov 02 01:53:46.635816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.635782    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-07ac70d0-openshift-default-dockercfg-bfbgl\""
Nov 02 01:53:46.662323 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662277    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-socket\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662554 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662341    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-data\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662554 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662375    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfgw\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-kube-api-access-rbfgw\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662554 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662397    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-token\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662554 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662424    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-credential-socket\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662571    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-podinfo\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662869 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662828    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-socket\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662924 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662833    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-envoy\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662924 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662893    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-credential-socket\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.662924 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.662916    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istiod-ca-cert\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.663084 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.663040    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-data\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.663142 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.663018    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-certs\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.663427 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.663390    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-certs\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.663816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.663792    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istiod-ca-cert\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.665730 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.665703    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-envoy\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.666180 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.666160    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-podinfo\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.753636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.736391    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-token\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.769762 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.761846    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfgw\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-kube-api-access-rbfgw\") pod \"e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") " pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:46.857274 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:46.856956    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:47.326649 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:47.326594    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"]
Nov 02 01:53:47.327717 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:53:47.327682    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd592bd_25cf_4ae9_9cb2_ab2ba12335db.slice/crio-bbdf46d913101a1e874a1787d5e6bcdb46fe039aa623e1e85d668c12d784129f WatchSource:0}: Error finding container bbdf46d913101a1e874a1787d5e6bcdb46fe039aa623e1e85d668c12d784129f: Status 404 returned error can't find the container with id bbdf46d913101a1e874a1787d5e6bcdb46fe039aa623e1e85d668c12d784129f
Nov 02 01:53:47.330947 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:47.330904    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:53:47.331076 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:47.331026    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:53:47.331148 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:47.331080    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:53:47.445245 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:47.445154    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" event={"ID":"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db","Type":"ContainerStarted","Data":"60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9"}
Nov 02 01:53:47.445245 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:47.445215    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" event={"ID":"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db","Type":"ContainerStarted","Data":"bbdf46d913101a1e874a1787d5e6bcdb46fe039aa623e1e85d668c12d784129f"}
Nov 02 01:53:47.566823 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:47.566761    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" podStartSLOduration=1.566744161 podStartE2EDuration="1.566744161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:53:47.564503886 +0000 UTC m=+1735.438144938" watchObservedRunningTime="2026-11-02 01:53:47.566744161 +0000 UTC m=+1735.440385212"
Nov 02 01:53:47.859863 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:47.859823    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:48.868102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:48.866756    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:49.569933 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:49.569893    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:53:49.571175 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:53:49.571153    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:54:13.889522 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:13.889490    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"]
Nov 02 01:54:13.889972 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:13.889733    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy" containerID="cri-o://be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49" gracePeriod=30
Nov 02 01:54:16.043658 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:54:16.043600    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:54:19.224517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.224482    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:54:19.295346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295307    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-socket\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.295532 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295413    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-data\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.295532 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295479    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-credential-socket\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.295532 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295517    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-kube-api-access-szf2z\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-kube-api-access-szf2z\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.295749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295594    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-certs\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.295749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295692    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4058d510-3481-400b-bb9a-6c9454ff1cef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4058d510-3481-400b-bb9a-6c9454ff1cef-istiod-ca-cert\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.295749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295706    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-data" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:19.295749 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295721    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-socket" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:19.295951 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295788    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-token\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-token\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.295951 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295829    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-podinfo\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.295951 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295843    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-credential-socket" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:19.296105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.295950    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-envoy\") pod \"4058d510-3481-400b-bb9a-6c9454ff1cef\" (UID: \"4058d510-3481-400b-bb9a-6c9454ff1cef\") "
Nov 02 01:54:19.296105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.296040    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-certs" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:19.296105 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.296071    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4058d510-3481-400b-bb9a-6c9454ff1cef-istiod-ca-cert" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:19.297843 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.297816    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.297843 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.297844    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.298080 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.297872    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.298080 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.297895    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.298080 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.297910    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4058d510-3481-400b-bb9a-6c9454ff1cef-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.298922 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.298894    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-token" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:19.299145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.299120    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-podinfo" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:19.299742 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.299719    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-envoy" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:19.300360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.300337    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-kube-api-access-szf2z" pod "4058d510-3481-400b-bb9a-6c9454ff1cef" (UID: "4058d510-3481-400b-bb9a-6c9454ff1cef"). InnerVolumeSpecName "kube-api-access-szf2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:19.414353 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.414272    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szf2z\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-kube-api-access-szf2z\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.414353 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.414336    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.414353 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.414352    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.414696 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.414366    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4058d510-3481-400b-bb9a-6c9454ff1cef-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:19.840317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.840278    4287 generic.go:356] "Generic (PLEG): container finished" podID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerID="be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49" exitCode=0
Nov 02 01:54:19.840678 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.840481    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" event={"ID":"4058d510-3481-400b-bb9a-6c9454ff1cef","Type":"ContainerDied","Data":"be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49"}
Nov 02 01:54:19.840780 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.840715    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7" event={"ID":"4058d510-3481-400b-bb9a-6c9454ff1cef","Type":"ContainerDied","Data":"401a7668ac3a71c4d801e57e32c433a4946e787023b6cd1b5aeab111907212e1"}
Nov 02 01:54:19.840780 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.840747    4287 scope.go:122] "RemoveContainer" containerID="be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49"
Nov 02 01:54:19.842412 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.841570    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"
Nov 02 01:54:19.869080 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.864328    4287 scope.go:122] "RemoveContainer" containerID="be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49"
Nov 02 01:54:19.869080 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:54:19.864769    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49\": container with ID starting with be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49 not found: ID does not exist" containerID="be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49"
Nov 02 01:54:19.869080 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:19.864823    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49"} err="failed to get container status \"be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49\": rpc error: code = NotFound desc = could not find container \"be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49\": container with ID starting with be7dce068a3a479230038610f5692b75d90a8f6fc97a9c447060720688d2bd49 not found: ID does not exist"
Nov 02 01:54:20.012345 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:20.012312    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"]
Nov 02 01:54:20.059174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:20.059124    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hxhw7"]
Nov 02 01:54:20.240487 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:20.240390    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" path="/var/lib/kubelet/pods/4058d510-3481-400b-bb9a-6c9454ff1cef/volumes"
Nov 02 01:54:42.565631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:42.562209    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"]
Nov 02 01:54:42.571634 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:42.570169    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" containerID="cri-o://25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609" gracePeriod=30
Nov 02 01:54:46.056553 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:54:46.056497    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:54:46.586534 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:46.586499    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:54:46.586735 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:46.586564    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:54:47.887303 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.887272    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:54:47.948630 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.948581    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-socket\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.948820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.948662    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-credential-socket\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.948820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.948729    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-certs\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.948820 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.948803    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-token\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-token\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.948998 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.948829    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-podinfo\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.948998 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.948886    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-socket" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:47.948998 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.948905    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-data\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.948998 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.948974    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-kube-api-access-wb8tx\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-kube-api-access-wb8tx\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.949203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.949016    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-certs" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:47.949203 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.949179    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-data" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:47.949315 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.949185    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3435e444-4c0f-441b-a1c3-8623ce6b05db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3435e444-4c0f-441b-a1c3-8623ce6b05db-istiod-ca-cert\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.949315 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.949257    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-envoy\") pod \"3435e444-4c0f-441b-a1c3-8623ce6b05db\" (UID: \"3435e444-4c0f-441b-a1c3-8623ce6b05db\") "
Nov 02 01:54:47.949589 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.949505    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3435e444-4c0f-441b-a1c3-8623ce6b05db-istiod-ca-cert" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:54:47.949909 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.949858    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-credential-socket" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:47.950248 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.950226    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:47.950366 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.950254    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3435e444-4c0f-441b-a1c3-8623ce6b05db-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:47.950366 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.950270    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:47.950366 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.950283    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:47.950366 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.950300    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:47.951597 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.951573    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-podinfo" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:54:47.951755 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.951600    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-kube-api-access-wb8tx" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "kube-api-access-wb8tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:47.951935 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.951918    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-token" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:54:47.952165 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:47.952141    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-envoy" pod "3435e444-4c0f-441b-a1c3-8623ce6b05db" (UID: "3435e444-4c0f-441b-a1c3-8623ce6b05db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:54:48.051928 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.051893    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb8tx\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-kube-api-access-wb8tx\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:48.051928 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.051923    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:48.051928 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.051933    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:48.052169 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.051941    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3435e444-4c0f-441b-a1c3-8623ce6b05db-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:54:48.084666 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.077686    4287 generic.go:356] "Generic (PLEG): container finished" podID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerID="25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609" exitCode=0
Nov 02 01:54:48.084666 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.077743    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"
Nov 02 01:54:48.084666 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.077740    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" event={"ID":"3435e444-4c0f-441b-a1c3-8623ce6b05db","Type":"ContainerDied","Data":"25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609"}
Nov 02 01:54:48.084666 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.077850    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9" event={"ID":"3435e444-4c0f-441b-a1c3-8623ce6b05db","Type":"ContainerDied","Data":"f74207ae52a8373282913582e6980cdd9ff4fd934f7c125d42f0273a75aa79dc"}
Nov 02 01:54:48.084666 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.077889    4287 scope.go:122] "RemoveContainer" containerID="25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609"
Nov 02 01:54:48.093677 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.093654    4287 scope.go:122] "RemoveContainer" containerID="25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609"
Nov 02 01:54:48.093996 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:54:48.093977    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609\": container with ID starting with 25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609 not found: ID does not exist" containerID="25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609"
Nov 02 01:54:48.094083 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.094011    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609"} err="failed to get container status \"25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609\": rpc error: code = NotFound desc = could not find container \"25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609\": container with ID starting with 25ef8b8f535a775473fbb2914e0af7c18bc18cb40417a6cbbacb49e6f2997609 not found: ID does not exist"
Nov 02 01:54:48.248941 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.248868    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"]
Nov 02 01:54:48.252772 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:48.252746    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-qssx9"]
Nov 02 01:54:50.238206 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:50.238170    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" path="/var/lib/kubelet/pods/3435e444-4c0f-441b-a1c3-8623ce6b05db/volumes"
Nov 02 01:54:52.479841 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:52.479812    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:54:52.503589 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:54:52.503561    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:55:13.146317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:13.146281    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"]
Nov 02 01:55:13.146851 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:13.146507    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy" containerID="cri-o://6a2bbe5c98ea9103d7f78c01bf5b4f2498b22aa8294a174696825a1f386508a2" gracePeriod=30
Nov 02 01:55:14.268509 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:14.268474    4287 generic.go:356] "Generic (PLEG): container finished" podID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerID="60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9" exitCode=0
Nov 02 01:55:14.268964 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:14.268543    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" event={"ID":"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db","Type":"ContainerDied","Data":"60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9"}
Nov 02 01:55:14.268964 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:14.268953    4287 scope.go:122] "RemoveContainer" containerID="60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9"
Nov 02 01:55:14.270534 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:14.270507    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:14.270628 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:14.270572    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:14.270682 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:14.270634    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:55:15.282068 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:15.282031    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" event={"ID":"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db","Type":"ContainerStarted","Data":"521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce"}
Nov 02 01:55:15.869185 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:15.861113    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:55:15.869185 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:15.868313    4287 patch_prober.go:29] interesting pod/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:55:15.869185 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:15.868381    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 01:55:16.039216 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:55:16.039178    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:55:16.871636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:16.866150    4287 patch_prober.go:29] interesting pod/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:55:16.871636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:16.866254    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 01:55:17.665644 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:17.665485    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:55:17.665644 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:17.665545    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:55:17.870643 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:17.869982    4287 patch_prober.go:29] interesting pod/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:55:17.870643 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:17.870047    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 01:55:18.304960 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.304927    4287 generic.go:356] "Generic (PLEG): container finished" podID="67644676-76e5-48ca-8542-9dc6af47e394" containerID="6a2bbe5c98ea9103d7f78c01bf5b4f2498b22aa8294a174696825a1f386508a2" exitCode=0
Nov 02 01:55:18.305332 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.305092    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" event={"ID":"67644676-76e5-48ca-8542-9dc6af47e394","Type":"ContainerDied","Data":"6a2bbe5c98ea9103d7f78c01bf5b4f2498b22aa8294a174696825a1f386508a2"}
Nov 02 01:55:18.487007 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.486980    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:55:18.514893 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.514858    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67644676-76e5-48ca-8542-9dc6af47e394-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644676-76e5-48ca-8542-9dc6af47e394-istio-podinfo\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.515276 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515259    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67644676-76e5-48ca-8542-9dc6af47e394-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644676-76e5-48ca-8542-9dc6af47e394-istiod-ca-cert\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.515361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515291    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-kube-api-access-rp8rp\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-kube-api-access-rp8rp\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.515361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515318    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-socket\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.515361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515354    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-certs\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.515493 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515424    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-data\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.515493 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515446    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-istio-token\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-istio-token\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.515558 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515533    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-credential-socket\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.515652 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515638    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-envoy\") pod \"67644676-76e5-48ca-8542-9dc6af47e394\" (UID: \"67644676-76e5-48ca-8542-9dc6af47e394\") "
Nov 02 01:55:18.516146 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515730    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67644676-76e5-48ca-8542-9dc6af47e394-istiod-ca-cert" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:55:18.516146 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.515978    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-credential-socket" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.516146 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.516032    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-data" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.516402 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.516339    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.516402 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.516363    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.516402 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.516380    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67644676-76e5-48ca-8542-9dc6af47e394-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.516797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.516735    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-socket" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.516797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.516785    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-certs" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.517549 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.517511    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67644676-76e5-48ca-8542-9dc6af47e394-istio-podinfo" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:55:18.518406 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.518378    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-istio-token" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:18.518917 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.518894    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-envoy" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:55:18.519101 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.519083    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-kube-api-access-rp8rp" pod "67644676-76e5-48ca-8542-9dc6af47e394" (UID: "67644676-76e5-48ca-8542-9dc6af47e394"). InnerVolumeSpecName "kube-api-access-rp8rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:55:18.618708 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.618675    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67644676-76e5-48ca-8542-9dc6af47e394-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.618708 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.618704    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp8rp\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-kube-api-access-rp8rp\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.619179 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.618721    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.619179 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.618732    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.619179 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.618740    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67644676-76e5-48ca-8542-9dc6af47e394-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.619179 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.618750    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67644676-76e5-48ca-8542-9dc6af47e394-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:55:18.875126 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.875081    4287 patch_prober.go:29] interesting pod/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:55:18.875344 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:18.875148    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 01:55:19.313264 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:19.313232    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"
Nov 02 01:55:19.313764 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:19.313266    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6" event={"ID":"67644676-76e5-48ca-8542-9dc6af47e394","Type":"ContainerDied","Data":"2777acfb1ce61b4419edde8a637ba0ceeab2ff20f6e992c143db09e88aa8b731"}
Nov 02 01:55:19.313764 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:19.313343    4287 scope.go:122] "RemoveContainer" containerID="6a2bbe5c98ea9103d7f78c01bf5b4f2498b22aa8294a174696825a1f386508a2"
Nov 02 01:55:19.546343 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:19.546300    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"]
Nov 02 01:55:19.582833 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:19.582750    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:55:19.584290 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:19.584264    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-qgrp6"]
Nov 02 01:55:19.873846 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:19.873764    4287 patch_prober.go:29] interesting pod/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 01:55:19.874067 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:19.874046    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 01:55:20.238707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:20.238630    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67644676-76e5-48ca-8542-9dc6af47e394" path="/var/lib/kubelet/pods/67644676-76e5-48ca-8542-9dc6af47e394/volumes"
Nov 02 01:55:20.865132 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:20.865075    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:55:20.866535 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:55:20.866510    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 01:55:46.039307 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:55:46.039270    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:56:16.040341 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:56:16.040306    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:56:46.040270 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:56:46.040227    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:56:49.209845 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.209812    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"]
Nov 02 01:56:49.210319 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.210001    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47" podUID="8ccfadf3-2ab6-4783-ab01-ebfdff800afb" containerName="maas-api" containerID="cri-o://b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736" gracePeriod=30
Nov 02 01:56:49.586483 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.586452    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:56:49.663638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.657998    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-openshift-service-ca" pod "8ccfadf3-2ab6-4783-ab01-ebfdff800afb" (UID: "8ccfadf3-2ab6-4783-ab01-ebfdff800afb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:56:49.663638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.658221    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-openshift-service-ca\") pod \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") "
Nov 02 01:56:49.663638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.658421    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls\") pod \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") "
Nov 02 01:56:49.663638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.659165    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-metrics-certs\") pod \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") "
Nov 02 01:56:49.663638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.659782    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-kube-api-access-tmm5l\" (UniqueName: \"kubernetes.io/projected/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-kube-api-access-tmm5l\") pod \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\" (UID: \"8ccfadf3-2ab6-4783-ab01-ebfdff800afb\") "
Nov 02 01:56:49.663638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.660511    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:49.686640 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.686587    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-kube-api-access-tmm5l" pod "8ccfadf3-2ab6-4783-ab01-ebfdff800afb" (UID: "8ccfadf3-2ab6-4783-ab01-ebfdff800afb"). InnerVolumeSpecName "kube-api-access-tmm5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:56:49.696800 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.696755    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls" pod "8ccfadf3-2ab6-4783-ab01-ebfdff800afb" (UID: "8ccfadf3-2ab6-4783-ab01-ebfdff800afb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:49.709844 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.702783    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-metrics-certs" pod "8ccfadf3-2ab6-4783-ab01-ebfdff800afb" (UID: "8ccfadf3-2ab6-4783-ab01-ebfdff800afb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:56:49.764985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.764904    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:49.764985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.764941    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:49.764985 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:49.764957    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmm5l\" (UniqueName: \"kubernetes.io/projected/8ccfadf3-2ab6-4783-ab01-ebfdff800afb-kube-api-access-tmm5l\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:56:50.337350 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.337234    4287 generic.go:356] "Generic (PLEG): container finished" podID="8ccfadf3-2ab6-4783-ab01-ebfdff800afb" containerID="b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736" exitCode=0
Nov 02 01:56:50.337350 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.337320    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"
Nov 02 01:56:50.337773 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.337506    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47" event={"ID":"8ccfadf3-2ab6-4783-ab01-ebfdff800afb","Type":"ContainerDied","Data":"b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736"}
Nov 02 01:56:50.337773 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.337559    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47" event={"ID":"8ccfadf3-2ab6-4783-ab01-ebfdff800afb","Type":"ContainerDied","Data":"c6925f31027d944b91f18556a2b979f2d48a51d613882db8573ad5ab1c647588"}
Nov 02 01:56:50.337773 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.337592    4287 scope.go:122] "RemoveContainer" containerID="b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736"
Nov 02 01:56:50.359455 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.359413    4287 scope.go:122] "RemoveContainer" containerID="b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736"
Nov 02 01:56:50.359919 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:56:50.359887    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736\": container with ID starting with b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736 not found: ID does not exist" containerID="b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736"
Nov 02 01:56:50.360089 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.359930    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736"} err="failed to get container status \"b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736\": rpc error: code = NotFound desc = could not find container \"b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736\": container with ID starting with b3a33d9907845796725da213898e0416b44c6e0cf5877af7b4f4e87a716fb736 not found: ID does not exist"
Nov 02 01:56:50.510418 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.510380    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"]
Nov 02 01:56:50.558971 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:50.558928    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-59842dd7-5dc95dbcc9-76q47"]
Nov 02 01:56:52.240227 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:56:52.240192    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ccfadf3-2ab6-4783-ab01-ebfdff800afb" path="/var/lib/kubelet/pods/8ccfadf3-2ab6-4783-ab01-ebfdff800afb/volumes"
Nov 02 01:57:13.734108 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:13.734069    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"]
Nov 02 01:57:13.734627 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:13.734357    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy" containerID="cri-o://f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802" gracePeriod=30
Nov 02 01:57:14.311360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:14.311325    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"]
Nov 02 01:57:14.311581 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:14.311553    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy" containerID="cri-o://99e1809f9f7c62bf863762dc1e2fcdd411ab540845eb2e5cd9de86b3b89cdaa3" gracePeriod=30
Nov 02 01:57:14.701414 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:14.701330    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"]
Nov 02 01:57:14.701561 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:14.701524    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy" containerID="cri-o://34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085" gracePeriod=30
Nov 02 01:57:15.002347 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:15.002266    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"]
Nov 02 01:57:15.002816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:15.002499    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy" containerID="cri-o://fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a" gracePeriod=30
Nov 02 01:57:15.008658 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:15.008597    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"]
Nov 02 01:57:15.009057 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:15.009023    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" containerID="cri-o://7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab" gracePeriod=30
Nov 02 01:57:16.039319 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:57:16.039280    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:57:17.275210 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:17.275172    4287 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:57:17.275776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:17.275245    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:57:18.761286 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:18.761254    4287 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 01:57:18.761587 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:18.761323    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 01:57:19.004926 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.004861    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:57:19.081139 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081109    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-certs\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081314 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081151    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84e4245e-60d6-4175-985a-5471f6fa790e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4245e-60d6-4175-985a-5471f6fa790e-istio-podinfo\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081314 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081183    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-istio-token\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-istio-token\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081345    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-envoy\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081389    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-socket\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081429 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081408    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-certs" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.081599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081460    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e4245e-60d6-4175-985a-5471f6fa790e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4245e-60d6-4175-985a-5471f6fa790e-istiod-ca-cert\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081516    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-data\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081599 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081562    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-credential-socket\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081710    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-kube-api-access-9w9v9\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-kube-api-access-9w9v9\") pod \"84e4245e-60d6-4175-985a-5471f6fa790e\" (UID: \"84e4245e-60d6-4175-985a-5471f6fa790e\") "
Nov 02 01:57:19.081793 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081749    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-socket" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.081904 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081806    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-data" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.081963 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.081897    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-credential-socket" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.082297 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.082273    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e4245e-60d6-4175-985a-5471f6fa790e-istiod-ca-cert" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:19.082534 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.082512    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.082629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.082542    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e4245e-60d6-4175-985a-5471f6fa790e-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.082629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.082557    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.082629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.082572    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.082629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.082584    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.083712 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.083688    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84e4245e-60d6-4175-985a-5471f6fa790e-istio-podinfo" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:19.083816 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.083781    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-envoy" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.083865 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.083826    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-istio-token" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:19.084004 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.083990    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-kube-api-access-9w9v9" pod "84e4245e-60d6-4175-985a-5471f6fa790e" (UID: "84e4245e-60d6-4175-985a-5471f6fa790e"). InnerVolumeSpecName "kube-api-access-9w9v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:19.184549 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.184517    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e4245e-60d6-4175-985a-5471f6fa790e-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.184549 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.184548    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w9v9\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-kube-api-access-9w9v9\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.184774 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.184558    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e4245e-60d6-4175-985a-5471f6fa790e-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.184774 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.184568    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e4245e-60d6-4175-985a-5471f6fa790e-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.548707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.548670    4287 generic.go:356] "Generic (PLEG): container finished" podID="57019be7-db39-44a3-8807-b65b7d577328" containerID="99e1809f9f7c62bf863762dc1e2fcdd411ab540845eb2e5cd9de86b3b89cdaa3" exitCode=0
Nov 02 01:57:19.548894 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.548738    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" event={"ID":"57019be7-db39-44a3-8807-b65b7d577328","Type":"ContainerDied","Data":"99e1809f9f7c62bf863762dc1e2fcdd411ab540845eb2e5cd9de86b3b89cdaa3"}
Nov 02 01:57:19.550677 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.550568    4287 generic.go:356] "Generic (PLEG): container finished" podID="84e4245e-60d6-4175-985a-5471f6fa790e" containerID="f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802" exitCode=0
Nov 02 01:57:19.550801 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.550778    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" event={"ID":"84e4245e-60d6-4175-985a-5471f6fa790e","Type":"ContainerDied","Data":"f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802"}
Nov 02 01:57:19.550842 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.550815    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw" event={"ID":"84e4245e-60d6-4175-985a-5471f6fa790e","Type":"ContainerDied","Data":"071b74d116f9d4a00d3f929cc0f05358c0fe125b2b6a9c02d1d772e5f9e91e5b"}
Nov 02 01:57:19.550842 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.550836    4287 scope.go:122] "RemoveContainer" containerID="f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802"
Nov 02 01:57:19.550979 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.550966    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"
Nov 02 01:57:19.564821 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.564801    4287 scope.go:122] "RemoveContainer" containerID="f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802"
Nov 02 01:57:19.565135 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:57:19.565097    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802\": container with ID starting with f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802 not found: ID does not exist" containerID="f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802"
Nov 02 01:57:19.565265 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.565144    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802"} err="failed to get container status \"f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802\": rpc error: code = NotFound desc = could not find container \"f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802\": container with ID starting with f2499fcd16fa8d8913df84cafbae037cb15cb2fea0f8902e70c706ed69d66802 not found: ID does not exist"
Nov 02 01:57:19.577067 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.577016    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:57:19.692211 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692171    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-envoy\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692231    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-kube-api-access-52zjt\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-kube-api-access-52zjt\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692416 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692312    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57019be7-db39-44a3-8807-b65b7d577328-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57019be7-db39-44a3-8807-b65b7d577328-istio-podinfo\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692539 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692432    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-data\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692539 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692480    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-credential-socket\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692678 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692538    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-socket\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692678 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692566    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-certs\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692678 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692664    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-istio-token\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-istio-token\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692838 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692757    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57019be7-db39-44a3-8807-b65b7d577328-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57019be7-db39-44a3-8807-b65b7d577328-istiod-ca-cert\") pod \"57019be7-db39-44a3-8807-b65b7d577328\" (UID: \"57019be7-db39-44a3-8807-b65b7d577328\") "
Nov 02 01:57:19.692838 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692758    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-data" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.692838 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692808    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-credential-socket" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.693017 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.692993    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-certs" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.693089 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.693066    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-socket" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.693348 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.693330    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.693434 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.693355    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.693434 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.693368    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.693434 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.693381    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.693434 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.693376    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57019be7-db39-44a3-8807-b65b7d577328-istiod-ca-cert" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:19.694729 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.694710    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57019be7-db39-44a3-8807-b65b7d577328-istio-podinfo" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:19.694867 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.694848    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-istio-token" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:19.695102 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.695084    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-envoy" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:19.695145 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.695100    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-kube-api-access-52zjt" pod "57019be7-db39-44a3-8807-b65b7d577328" (UID: "57019be7-db39-44a3-8807-b65b7d577328"). InnerVolumeSpecName "kube-api-access-52zjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:19.719432 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.719397    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"]
Nov 02 01:57:19.768898 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.768862    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-8g6xw"]
Nov 02 01:57:19.794829 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.794802    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.794829 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.794828    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57019be7-db39-44a3-8807-b65b7d577328-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.795005 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.794841    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57019be7-db39-44a3-8807-b65b7d577328-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.795005 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.794851    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zjt\" (UniqueName: \"kubernetes.io/projected/57019be7-db39-44a3-8807-b65b7d577328-kube-api-access-52zjt\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.795005 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.794860    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57019be7-db39-44a3-8807-b65b7d577328-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:19.982479 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:19.982452    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:57:20.105084 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.104977    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec77c94b-73bd-4608-be47-ec0997c04d8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec77c94b-73bd-4608-be47-ec0997c04d8b-istiod-ca-cert\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105084 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105003    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec77c94b-73bd-4608-be47-ec0997c04d8b-istiod-ca-cert" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:20.105361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105117    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-podinfo\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105169    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-kube-api-access-phm46\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-kube-api-access-phm46\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105206    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-socket\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105232    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-token\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-token\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105361 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105293    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-certs\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105407    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-envoy\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105433    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-data\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105505    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-credential-socket\") pod \"ec77c94b-73bd-4608-be47-ec0997c04d8b\" (UID: \"ec77c94b-73bd-4608-be47-ec0997c04d8b\") "
Nov 02 01:57:20.105857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105649    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-socket" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.105857 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105839    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-certs" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.106128 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.105938    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-data" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.106256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.106235    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-credential-socket" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.108927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.108226    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-kube-api-access-phm46" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "kube-api-access-phm46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.108927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.108550    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phm46\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-kube-api-access-phm46\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.108927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.108568    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.108927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.108584    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.108927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.108617    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.108927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.108810    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.108927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.108824    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec77c94b-73bd-4608-be47-ec0997c04d8b-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.108927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.108886    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-envoy" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.125995 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.111704    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-podinfo" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:20.125995 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.117375    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-token" pod "ec77c94b-73bd-4608-be47-ec0997c04d8b" (UID: "ec77c94b-73bd-4608-be47-ec0997c04d8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.210122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.210081    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.210122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.210123    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.210386 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.210137    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec77c94b-73bd-4608-be47-ec0997c04d8b-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.240593 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.240549    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" path="/var/lib/kubelet/pods/84e4245e-60d6-4175-985a-5471f6fa790e/volumes"
Nov 02 01:57:20.299656 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.299628    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:57:20.337122 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.337096    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:57:20.412719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412682    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-kube-api-access-r259r\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-kube-api-access-r259r\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.412719 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412723    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-socket\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.412961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412764    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/301a31fd-361d-4f0b-b3c6-e46374bd5094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301a31fd-361d-4f0b-b3c6-e46374bd5094-istiod-ca-cert\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.412961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412817    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-credential-socket\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.412961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412844    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-certs\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.412961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412872    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istiod-ca-cert\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.412961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412912    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-envoy\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.412961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412939    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-credential-socket\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.413283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.412970    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-podinfo\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.413283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413010    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-data\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.413283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413098    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-socket\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.413283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413124    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-credential-socket" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.413283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413132    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-socket" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.413283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413161    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-token\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-token\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.413283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413219    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-certs" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.413283 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413246    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-certs\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413304    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-data\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413356    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-envoy\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413366    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-data" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413391    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-podinfo\") pod \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\" (UID: \"0bc7e3ec-0c80-45a6-9bae-03108eb0b873\") "
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413313    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istiod-ca-cert" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413443    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-kube-api-access-w45fn\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-kube-api-access-w45fn\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413522    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-token\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-token\") pod \"301a31fd-361d-4f0b-b3c6-e46374bd5094\" (UID: \"301a31fd-361d-4f0b-b3c6-e46374bd5094\") "
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413555    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-data" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413655    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-socket" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.413738 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413668    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301a31fd-361d-4f0b-b3c6-e46374bd5094-istiod-ca-cert" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:57:20.414252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413940    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-credential-socket" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.414252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.413963    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-certs" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.414252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414178    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.414252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414208    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.414252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414225    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/301a31fd-361d-4f0b-b3c6-e46374bd5094-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.414252 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414239    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.415596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414485    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.415596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414552    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.415596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414677    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.415596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414690    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.415596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414704    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.415596 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.414717    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.416050 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.416021    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-kube-api-access-r259r" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "kube-api-access-r259r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.416415 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.416377    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-envoy" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.416622 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.416577    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-envoy" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:57:20.416703 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.416579    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-kube-api-access-w45fn" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "kube-api-access-w45fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.416874 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.416857    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-podinfo" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:20.417032 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.417009    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-podinfo" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 01:57:20.417137 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.417046    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-token" pod "301a31fd-361d-4f0b-b3c6-e46374bd5094" (UID: "301a31fd-361d-4f0b-b3c6-e46374bd5094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.417378 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.417364    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-token" pod "0bc7e3ec-0c80-45a6-9bae-03108eb0b873" (UID: "0bc7e3ec-0c80-45a6-9bae-03108eb0b873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:57:20.516062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.516025    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.516062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.516063    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.516062 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.516079    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.516324 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.516093    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45fn\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-kube-api-access-w45fn\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.516324 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.516106    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.516324 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.516117    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r259r\" (UniqueName: \"kubernetes.io/projected/0bc7e3ec-0c80-45a6-9bae-03108eb0b873-kube-api-access-r259r\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.516324 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.516129    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.516324 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.516143    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/301a31fd-361d-4f0b-b3c6-e46374bd5094-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:57:20.562821 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.562784    4287 generic.go:356] "Generic (PLEG): container finished" podID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerID="7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab" exitCode=0
Nov 02 01:57:20.563014 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.562840    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"
Nov 02 01:57:20.563014 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.562868    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" event={"ID":"301a31fd-361d-4f0b-b3c6-e46374bd5094","Type":"ContainerDied","Data":"7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab"}
Nov 02 01:57:20.563014 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.562929    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829" event={"ID":"301a31fd-361d-4f0b-b3c6-e46374bd5094","Type":"ContainerDied","Data":"b6781e911d544a0183a48292831148986fabba864797b32364c96a251dbf13cd"}
Nov 02 01:57:20.563014 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.562958    4287 scope.go:122] "RemoveContainer" containerID="7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab"
Nov 02 01:57:20.564685 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.564662    4287 generic.go:356] "Generic (PLEG): container finished" podID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerID="fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a" exitCode=0
Nov 02 01:57:20.564797 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.564775    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"
Nov 02 01:57:20.564865 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.564779    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" event={"ID":"0bc7e3ec-0c80-45a6-9bae-03108eb0b873","Type":"ContainerDied","Data":"fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a"}
Nov 02 01:57:20.564919 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.564898    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk" event={"ID":"0bc7e3ec-0c80-45a6-9bae-03108eb0b873","Type":"ContainerDied","Data":"020ab868259b5b015a6f8c258389cdca33155d82aa903445a501564422883aa5"}
Nov 02 01:57:20.573807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.573771    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2" event={"ID":"57019be7-db39-44a3-8807-b65b7d577328","Type":"ContainerDied","Data":"efbc9ce756c929e9f66ae0dee28dd383fb5108bddca1d5f0591b1808f897c525"}
Nov 02 01:57:20.573807 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.573803    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"
Nov 02 01:57:20.575552 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.575531    4287 scope.go:122] "RemoveContainer" containerID="7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab"
Nov 02 01:57:20.576019 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:57:20.575996    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab\": container with ID starting with 7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab not found: ID does not exist" containerID="7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab"
Nov 02 01:57:20.576118 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.576032    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab"} err="failed to get container status \"7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab\": rpc error: code = NotFound desc = could not find container \"7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab\": container with ID starting with 7edd5cda053176c7d69ee968bba8a79ee009f99f8f4bcae306b5f577864218ab not found: ID does not exist"
Nov 02 01:57:20.576118 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.576055    4287 scope.go:122] "RemoveContainer" containerID="fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a"
Nov 02 01:57:20.577054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.576909    4287 generic.go:356] "Generic (PLEG): container finished" podID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerID="34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085" exitCode=0
Nov 02 01:57:20.577054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.576934    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" event={"ID":"ec77c94b-73bd-4608-be47-ec0997c04d8b","Type":"ContainerDied","Data":"34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085"}
Nov 02 01:57:20.577054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.576967    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"
Nov 02 01:57:20.577054 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.576975    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5" event={"ID":"ec77c94b-73bd-4608-be47-ec0997c04d8b","Type":"ContainerDied","Data":"be09017a03b6b5f18a80f6b61a5690cb27ed241f91d8fcd602848b9ed27ffe04"}
Nov 02 01:57:20.587696 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.587669    4287 scope.go:122] "RemoveContainer" containerID="fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a"
Nov 02 01:57:20.588032 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:57:20.588001    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a\": container with ID starting with fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a not found: ID does not exist" containerID="fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a"
Nov 02 01:57:20.588128 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.588042    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a"} err="failed to get container status \"fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a\": rpc error: code = NotFound desc = could not find container \"fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a\": container with ID starting with fcf99d1d756197ffbc259f4a94eec27384351ce15c739974cbfa354b651ca27a not found: ID does not exist"
Nov 02 01:57:20.588128 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.588064    4287 scope.go:122] "RemoveContainer" containerID="99e1809f9f7c62bf863762dc1e2fcdd411ab540845eb2e5cd9de86b3b89cdaa3"
Nov 02 01:57:20.599623 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.599585    4287 scope.go:122] "RemoveContainer" containerID="34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085"
Nov 02 01:57:20.609129 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.609108    4287 scope.go:122] "RemoveContainer" containerID="34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085"
Nov 02 01:57:20.609405 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:57:20.609387    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085\": container with ID starting with 34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085 not found: ID does not exist" containerID="34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085"
Nov 02 01:57:20.609454 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.609416    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085"} err="failed to get container status \"34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085\": rpc error: code = NotFound desc = could not find container \"34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085\": container with ID starting with 34e6d8b019b30e682899cf06f9fe30a98aae1909c1d7de7c426f009434a75085 not found: ID does not exist"
Nov 02 01:57:20.730130 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.730094    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"]
Nov 02 01:57:20.733969 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.733939    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-sn7g2"]
Nov 02 01:57:20.845418 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.845357    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"]
Nov 02 01:57:20.857064 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.857030    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-l49tk"]
Nov 02 01:57:20.965833 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.965791    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"]
Nov 02 01:57:20.975389 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:20.975352    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-9m829"]
Nov 02 01:57:21.085027 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:21.084935    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"]
Nov 02 01:57:21.091369 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:21.091338    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-z57q5"]
Nov 02 01:57:22.238994 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:22.238959    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" path="/var/lib/kubelet/pods/0bc7e3ec-0c80-45a6-9bae-03108eb0b873/volumes"
Nov 02 01:57:22.239720 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:22.239698    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" path="/var/lib/kubelet/pods/301a31fd-361d-4f0b-b3c6-e46374bd5094/volumes"
Nov 02 01:57:22.240181 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:22.240160    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57019be7-db39-44a3-8807-b65b7d577328" path="/var/lib/kubelet/pods/57019be7-db39-44a3-8807-b65b7d577328/volumes"
Nov 02 01:57:22.240961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:22.240939    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" path="/var/lib/kubelet/pods/ec77c94b-73bd-4608-be47-ec0997c04d8b/volumes"
Nov 02 01:57:32.846813 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.846775    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"]
Nov 02 01:57:32.847295 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847283    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847296    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847308    4287 cpu_manager.go:414] "containerMap: removing container" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847314    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847320    4287 cpu_manager.go:414] "containerMap: removing container" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847325    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847333    4287 cpu_manager.go:414] "containerMap: removing container" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847338    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847347    4287 cpu_manager.go:414] "containerMap: removing container" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy"
Nov 02 01:57:32.847352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847352    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847362    4287 cpu_manager.go:414] "containerMap: removing container" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847366    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847377    4287 cpu_manager.go:414] "containerMap: removing container" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847382    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847389    4287 cpu_manager.go:414] "containerMap: removing container" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847395    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847404    4287 cpu_manager.go:414] "containerMap: removing container" podUID="8ccfadf3-2ab6-4783-ab01-ebfdff800afb" containerName="maas-api"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847408    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccfadf3-2ab6-4783-ab01-ebfdff800afb" containerName="maas-api"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847482    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3435e444-4c0f-441b-a1c3-8623ce6b05db" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847488    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec77c94b-73bd-4608-be47-ec0997c04d8b" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847495    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc7e3ec-0c80-45a6-9bae-03108eb0b873" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847501    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="57019be7-db39-44a3-8807-b65b7d577328" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847506    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e4245e-60d6-4175-985a-5471f6fa790e" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847511    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="4058d510-3481-400b-bb9a-6c9454ff1cef" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847517    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="301a31fd-361d-4f0b-b3c6-e46374bd5094" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847522    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="67644676-76e5-48ca-8542-9dc6af47e394" containerName="istio-proxy"
Nov 02 01:57:32.847629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.847527    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccfadf3-2ab6-4783-ab01-ebfdff800afb" containerName="maas-api"
Nov 02 01:57:32.852152 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.852137    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:32.905064 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.905033    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-39ceb2-serving-cert\""
Nov 02 01:57:32.958208 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.958173    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"]
Nov 02 01:57:32.980287 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.980257    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:32.980721 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.980690    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:32.980960 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.980940    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxh5x\" (UniqueName: \"kubernetes.io/projected/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-kube-api-access-zxh5x\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:32.981239 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:32.981222    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.082108 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.082053    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.082256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.082128    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.082256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.082176    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.082256 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.082213    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxh5x\" (UniqueName: \"kubernetes.io/projected/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-kube-api-access-zxh5x\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.082256 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:57:33.082232    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-serving-cert: secret "maas-api-e2e-shared-b-w5-39ceb2-serving-cert" not found
Nov 02 01:57:33.082436 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:57:33.082338    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls podName:a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b nodeName:}" failed. No retries permitted until 2026-11-02 01:57:33.582316172 +0000 UTC m=+1961.455957219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls") pod "maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b" (UID: "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b") : secret "maas-api-e2e-shared-b-w5-39ceb2-serving-cert" not found
Nov 02 01:57:33.082866 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.082839    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.084817 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.084798    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.144812 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.144735    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxh5x\" (UniqueName: \"kubernetes.io/projected/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-kube-api-access-zxh5x\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.588538 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.588503    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.591141 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.591113    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:33.766360 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:33.766322    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:34.153949 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:34.153917    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"]
Nov 02 01:57:34.155096 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:34.155072    4287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 01:57:34.670507 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:34.670477    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b" event={"ID":"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b","Type":"ContainerStarted","Data":"c915f07a07bea1b9635d091529a4db4acdd9951ac3114d926e2c1fbd6c317c4d"}
Nov 02 01:57:35.677910 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:35.677868    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b" event={"ID":"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b","Type":"ContainerStarted","Data":"17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09"}
Nov 02 01:57:35.678302 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:35.677937    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:35.810085 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:35.810028    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b" podStartSLOduration=2.9690512570000003 podStartE2EDuration="3.810011955s" totalImagesPullingTime="840.960698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:57:35.805791978 +0000 UTC m=+1963.679433030" watchObservedRunningTime="2026-11-02 01:57:35.810011955 +0000 UTC m=+1963.683653036"
Nov 02 01:57:41.690694 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:57:41.690664    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 01:57:46.039433 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:57:46.039396    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:58:13.798466 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:13.798427    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"]
Nov 02 01:58:13.803228 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:13.803210    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:13.906927 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:13.906888    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-19436b-serving-cert\""
Nov 02 01:58:13.914469 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:13.914429    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"]
Nov 02 01:58:13.923485 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:13.923452    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wxj2\" (UniqueName: \"kubernetes.io/projected/21073d91-773c-4ff8-9486-4b78a3c177a3-kube-api-access-8wxj2\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:13.923636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:13.923518    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:13.923636 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:13.923589    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:13.923811 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:13.923789    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21073d91-773c-4ff8-9486-4b78a3c177a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.025078 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.025040    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.025078 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.025085    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.025338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.025202    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21073d91-773c-4ff8-9486-4b78a3c177a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.025338 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:58:14.025223    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-serving-cert: secret "maas-api-e2e-shared-b-w6-19436b-serving-cert" not found
Nov 02 01:58:14.025338 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.025271    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxj2\" (UniqueName: \"kubernetes.io/projected/21073d91-773c-4ff8-9486-4b78a3c177a3-kube-api-access-8wxj2\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.025338 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:58:14.025314    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls podName:21073d91-773c-4ff8-9486-4b78a3c177a3 nodeName:}" failed. No retries permitted until 2026-11-02 01:58:14.525291192 +0000 UTC m=+2002.398932231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls") pod "maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4" (UID: "21073d91-773c-4ff8-9486-4b78a3c177a3") : secret "maas-api-e2e-shared-b-w6-19436b-serving-cert" not found
Nov 02 01:58:14.025892 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.025855    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21073d91-773c-4ff8-9486-4b78a3c177a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.027698 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.027677    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.089528 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.089446    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxj2\" (UniqueName: \"kubernetes.io/projected/21073d91-773c-4ff8-9486-4b78a3c177a3-kube-api-access-8wxj2\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.531313 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.531275    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.533939 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.533909    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:14.717352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:14.717318    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:15.097594 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:15.097567    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"]
Nov 02 01:58:15.097961 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:58:15.097779    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21073d91_773c_4ff8_9486_4b78a3c177a3.slice/crio-ad22b30816399afe78cad53372395b7080bfb41a9f386e8a4e74dc9a82220c56 WatchSource:0}: Error finding container ad22b30816399afe78cad53372395b7080bfb41a9f386e8a4e74dc9a82220c56: Status 404 returned error can't find the container with id ad22b30816399afe78cad53372395b7080bfb41a9f386e8a4e74dc9a82220c56
Nov 02 01:58:15.959982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:15.959953    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4" event={"ID":"21073d91-773c-4ff8-9486-4b78a3c177a3","Type":"ContainerStarted","Data":"ad22b30816399afe78cad53372395b7080bfb41a9f386e8a4e74dc9a82220c56"}
Nov 02 01:58:16.044167 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:58:16.044125    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:58:17.021287 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:17.021247    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4" event={"ID":"21073d91-773c-4ff8-9486-4b78a3c177a3","Type":"ContainerStarted","Data":"ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8"}
Nov 02 01:58:17.024771 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:17.022572    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:17.140307 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:17.140230    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4" podStartSLOduration=3.319779668 podStartE2EDuration="4.140211086s" totalImagesPullingTime="820.431418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:17.139434933 +0000 UTC m=+2005.013075987" watchObservedRunningTime="2026-11-02 01:58:17.140211086 +0000 UTC m=+2005.013852139"
Nov 02 01:58:24.041730 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:58:24.041702    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 01:58:46.038948 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:58:46.038908    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:59:05.166425 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.166386    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"]
Nov 02 01:59:05.166980 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.166623    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq" podUID="bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" containerName="discovery" containerID="cri-o://1b58e31f20269aa2cd680f8f724dee3fa6d9f7797bf2e3b466b938eece6b788f" gracePeriod=30
Nov 02 01:59:05.331982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.331947    4287 generic.go:356] "Generic (PLEG): container finished" podID="bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" containerID="1b58e31f20269aa2cd680f8f724dee3fa6d9f7797bf2e3b466b938eece6b788f" exitCode=0
Nov 02 01:59:05.332155 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.332018    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq" event={"ID":"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc","Type":"ContainerDied","Data":"1b58e31f20269aa2cd680f8f724dee3fa6d9f7797bf2e3b466b938eece6b788f"}
Nov 02 01:59:05.434257 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.434192    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:59:05.564251 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.564218    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-dns-cert\") pod \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") "
Nov 02 01:59:05.564251 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.564255    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-cacerts\") pod \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") "
Nov 02 01:59:05.564517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.564310    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-local-certs\") pod \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") "
Nov 02 01:59:05.564517 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.564423    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-ca-configmap\") pod \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") "
Nov 02 01:59:05.564676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.564573    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-kubeconfig\") pod \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") "
Nov 02 01:59:05.564676 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.564667    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-token\") pod \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") "
Nov 02 01:59:05.564865 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.564755    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-kube-api-access-2k8lb\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-kube-api-access-2k8lb\") pod \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\" (UID: \"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc\") "
Nov 02 01:59:05.564865 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.564822    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-ca-configmap" pod "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" (UID: "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 01:59:05.565284 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.565261    4287 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-ca-configmap\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.567276 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.567246    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-dns-cert" pod "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" (UID: "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:05.567409 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.567315    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-local-certs" pod "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" (UID: "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 01:59:05.567777 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.567732    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-kube-api-access-2k8lb" pod "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" (UID: "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc"). InnerVolumeSpecName "kube-api-access-2k8lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:05.568215 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.568021    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-cacerts" pod "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" (UID: "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:05.568215 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.568072    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-kubeconfig" pod "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" (UID: "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 01:59:05.568215 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.568189    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-token" pod "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" (UID: "bbfd0ee8-150f-44ad-bae5-52f7142ae3cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 01:59:05.666949 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.666917    4287 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-local-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.666949 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.666947    4287 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-kubeconfig\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.666949 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.666957    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.667183 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.666967    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8lb\" (UniqueName: \"kubernetes.io/projected/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-kube-api-access-2k8lb\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.667183 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.666977    4287 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-istio-csr-dns-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:05.667183 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:05.666986    4287 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc-cacerts\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 01:59:06.339374 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:06.339324    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq" event={"ID":"bbfd0ee8-150f-44ad-bae5-52f7142ae3cc","Type":"ContainerDied","Data":"e756d7b71fc31c479004acaf6fdb20029e220873166b4386db870d2ea489200d"}
Nov 02 01:59:06.339755 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:06.339406    4287 scope.go:122] "RemoveContainer" containerID="1b58e31f20269aa2cd680f8f724dee3fa6d9f7797bf2e3b466b938eece6b788f"
Nov 02 01:59:06.339755 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:06.339577    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"
Nov 02 01:59:06.555949 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:06.555916    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"]
Nov 02 01:59:06.605008 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:06.604980    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jmjhq"]
Nov 02 01:59:08.239001 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:08.238970    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" path="/var/lib/kubelet/pods/bbfd0ee8-150f-44ad-bae5-52f7142ae3cc/volumes"
Nov 02 01:59:12.787374 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.787336    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"]
Nov 02 01:59:12.787899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.787881    4287 cpu_manager.go:414] "containerMap: removing container" podUID="bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" containerName="discovery"
Nov 02 01:59:12.787953 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.787901    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" containerName="discovery"
Nov 02 01:59:12.788025 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.788016    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbfd0ee8-150f-44ad-bae5-52f7142ae3cc" containerName="discovery"
Nov 02 01:59:12.792631 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.792589    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.859420 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.859388    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-269235-openshift-default-dockercfg-b2x2z\""
Nov 02 01:59:12.967299 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.967259    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-envoy\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.967915 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.967887    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-socket\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.968052 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.967960    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c2e456-b896-4e5f-a696-7e26fabff35e-istiod-ca-cert\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.968111 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.968054    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-certs\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.968111 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.968083    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-podinfo\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.968217 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.968113    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-data\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.968217 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.968168    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-credential-socket\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.968315 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.968239    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-token\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.968315 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.968278    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz6fg\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-kube-api-access-vz6fg\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:12.997826 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:12.997676    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"]
Nov 02 01:59:13.069486 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069406    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-socket\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069486 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069470    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c2e456-b896-4e5f-a696-7e26fabff35e-istiod-ca-cert\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069763 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069528    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-certs\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069763 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069545    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-podinfo\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069763 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069644    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-data\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069763 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069711    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-credential-socket\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069780    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-token\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069833    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz6fg\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-kube-api-access-vz6fg\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069872    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-envoy\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069893    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-socket\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.069982 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.069959    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-certs\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.070249 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.070234    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-credential-socket\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.070365 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.070317    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-data\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.070365 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.070349    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c2e456-b896-4e5f-a696-7e26fabff35e-istiod-ca-cert\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.072545 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.072519    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-envoy\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.073092 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.073075    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-podinfo\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.138343 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.138302    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz6fg\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-kube-api-access-vz6fg\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.138707 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.138684    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-token\") pod \"e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") " pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.408972 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.408874    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:13.849859 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.849834    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"]
Nov 02 01:59:13.851227 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:59:13.851196    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c2e456_b896_4e5f_a696_7e26fabff35e.slice/crio-b04a230eefec660c6e25d7e648b75b07a715fc881c60b163a151baa7638bf127 WatchSource:0}: Error finding container b04a230eefec660c6e25d7e648b75b07a715fc881c60b163a151baa7638bf127: Status 404 returned error can't find the container with id b04a230eefec660c6e25d7e648b75b07a715fc881c60b163a151baa7638bf127
Nov 02 01:59:13.854016 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.853983    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:59:13.854119 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.854063    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:59:13.854119 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:13.854109    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:59:14.398342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:14.398298    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" event={"ID":"65c2e456-b896-4e5f-a696-7e26fabff35e","Type":"ContainerStarted","Data":"f92f99adb09d6d904fb6866957ee8727dc5ce16a8de874f88287583d0b4d40f2"}
Nov 02 01:59:14.398491 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:14.398382    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" event={"ID":"65c2e456-b896-4e5f-a696-7e26fabff35e","Type":"ContainerStarted","Data":"b04a230eefec660c6e25d7e648b75b07a715fc881c60b163a151baa7638bf127"}
Nov 02 01:59:14.409831 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:14.409813    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:14.415990 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:14.415970    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:14.518808 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:14.518767    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" podStartSLOduration=2.518752607 podStartE2EDuration="2.518752607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:14.517937957 +0000 UTC m=+2062.391579009" watchObservedRunningTime="2026-11-02 01:59:14.518752607 +0000 UTC m=+2062.392393660"
Nov 02 01:59:15.403967 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:15.403930    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:15.404906 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:15.404884    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 01:59:16.040807 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:59:16.040767    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:59:21.102046 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.102008    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"]
Nov 02 01:59:21.106874 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.106853    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.164638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.160189    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-69c15d-openshift-default-dockercfg-998jf\""
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.175495    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-socket\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.175577    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-token\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.175651    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-envoy\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.175719    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istiod-ca-cert\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.175875    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-data\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.175910    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-credential-socket\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.175962    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57cmj\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-kube-api-access-57cmj\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.175994    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-certs\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.176233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.176033    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-podinfo\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.238536 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.238502    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"]
Nov 02 01:59:21.305638 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.278440    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-credential-socket\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.306024 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.305989    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57cmj\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-kube-api-access-57cmj\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.306239 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.306219    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-certs\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.306404 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.306382    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-podinfo\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.306562 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.306534    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-socket\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.306832 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.306812    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-token\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.307027 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.307004    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-envoy\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.307238 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.307219    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istiod-ca-cert\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.307647 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.307624    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-data\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.324635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.308049    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-socket\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.324635 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.313592    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istiod-ca-cert\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.332253 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.332201    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-data\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.337961 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.337889    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-certs\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.338394 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.338342    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-credential-socket\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.339331 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.339181    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-podinfo\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.354733 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.354640    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-envoy\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.377984 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.377951    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-token\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.378312 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.378291    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57cmj\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-kube-api-access-57cmj\") pod \"e2e-inf-a-69c15d-openshift-default-5fb794595-642xz\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") " pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.421932 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.421891    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:21.896038 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.896014    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"]
Nov 02 01:59:21.897220 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:59:21.897192    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7415c203_cde7_40e1_a9a1_b6d3abaf7272.slice/crio-b0fad9eec2fbe77304e6710db52137f05c6d31f073ec3839e436a206005cc012 WatchSource:0}: Error finding container b0fad9eec2fbe77304e6710db52137f05c6d31f073ec3839e436a206005cc012: Status 404 returned error can't find the container with id b0fad9eec2fbe77304e6710db52137f05c6d31f073ec3839e436a206005cc012
Nov 02 01:59:21.900344 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.900313    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:59:21.900426 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.900373    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:59:21.900426 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:21.900399    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 01:59:22.468522 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:22.468484    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" event={"ID":"7415c203-cde7-40e1-a9a1-b6d3abaf7272","Type":"ContainerStarted","Data":"ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce"}
Nov 02 01:59:22.468962 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:22.468556    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" event={"ID":"7415c203-cde7-40e1-a9a1-b6d3abaf7272","Type":"ContainerStarted","Data":"b0fad9eec2fbe77304e6710db52137f05c6d31f073ec3839e436a206005cc012"}
Nov 02 01:59:22.590629 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:22.590544    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" podStartSLOduration=1.590530845 podStartE2EDuration="1.590530845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:22.589124562 +0000 UTC m=+2070.462765616" watchObservedRunningTime="2026-11-02 01:59:22.590530845 +0000 UTC m=+2070.464171897"
Nov 02 01:59:23.422188 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:23.422152    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:23.428750 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:23.428724    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:23.474342 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:23.474299    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:23.475243 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:23.475225    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 01:59:32.496287 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.496242    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"]
Nov 02 01:59:32.502352 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.502317    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.548456 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.548422    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/712e00b8-0705-43e4-ad47-a2e79e00e4fd-kube-api-access-5hnf8\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.548690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.548561    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.548690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.548653    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.548814 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.548708    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e00b8-0705-43e4-ad47-a2e79e00e4fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.555199 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.555171    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"]
Nov 02 01:59:32.600565 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.600525    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ef5a72-serving-cert\""
Nov 02 01:59:32.650304 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.650268    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.650500 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.650312    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.650500 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.650389    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e00b8-0705-43e4-ad47-a2e79e00e4fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.650500 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:59:32.650482    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-serving-cert: secret "maas-api-e2e-shared-b-w3-ef5a72-serving-cert" not found
Nov 02 01:59:32.650730 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.650561    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/712e00b8-0705-43e4-ad47-a2e79e00e4fd-kube-api-access-5hnf8\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.650730 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:59:32.650575    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls podName:712e00b8-0705-43e4-ad47-a2e79e00e4fd nodeName:}" failed. No retries permitted until 2026-11-02 01:59:33.15055892 +0000 UTC m=+2081.024199963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls") pod "maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn" (UID: "712e00b8-0705-43e4-ad47-a2e79e00e4fd") : secret "maas-api-e2e-shared-b-w3-ef5a72-serving-cert" not found
Nov 02 01:59:32.651081 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.651064    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e00b8-0705-43e4-ad47-a2e79e00e4fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.652898 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.652874    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:32.711819 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:32.711781    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/712e00b8-0705-43e4-ad47-a2e79e00e4fd-kube-api-access-5hnf8\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:33.157728 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:33.157684    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:33.160358 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:33.160323    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:33.417885 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:33.417794    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:33.797037 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:33.797010    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"]
Nov 02 01:59:33.797678 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:59:33.797652    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712e00b8_0705_43e4_ad47_a2e79e00e4fd.slice/crio-38be1516736176874f866e2246371a3955187648fa19795820ab1d434a8ac64d WatchSource:0}: Error finding container 38be1516736176874f866e2246371a3955187648fa19795820ab1d434a8ac64d: Status 404 returned error can't find the container with id 38be1516736176874f866e2246371a3955187648fa19795820ab1d434a8ac64d
Nov 02 01:59:34.545174 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:34.545142    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn" event={"ID":"712e00b8-0705-43e4-ad47-a2e79e00e4fd","Type":"ContainerStarted","Data":"38be1516736176874f866e2246371a3955187648fa19795820ab1d434a8ac64d"}
Nov 02 01:59:35.552776 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:35.552725    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn" event={"ID":"712e00b8-0705-43e4-ad47-a2e79e00e4fd","Type":"ContainerStarted","Data":"ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4"}
Nov 02 01:59:35.553213 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:35.552798    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:35.685690 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:35.685625    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn" podStartSLOduration=2.784368821 podStartE2EDuration="3.685585961s" totalImagesPullingTime="901.21714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:35.68492401 +0000 UTC m=+2083.558565063" watchObservedRunningTime="2026-11-02 01:59:35.685585961 +0000 UTC m=+2083.559227014"
Nov 02 01:59:41.564569 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:41.564539    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 01:59:46.039636 ip-10-0-134-183 kubenswrapper[4287]: E1102 01:59:46.039582    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 01:59:52.464317 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.464278    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"]
Nov 02 01:59:52.468511 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.468487    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.529190 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.529156    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:59:52.555977 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.555949    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 01:59:52.572233 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.572193    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c600fc-serving-cert\""
Nov 02 01:59:52.575899 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.575863    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"]
Nov 02 01:59:52.641390 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.641350    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.641555 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.641415    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-metrics-certs\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.641555 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.641545    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-maas-api-tls\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.641668 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.641638    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkq87\" (UniqueName: \"kubernetes.io/projected/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-kube-api-access-jkq87\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.742766 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.742680    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.742766 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.742749    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-metrics-certs\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.743065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.742821    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-maas-api-tls\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.743065 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.742867    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkq87\" (UniqueName: \"kubernetes.io/projected/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-kube-api-access-jkq87\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.743447 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.743424    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.745431 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.745410    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-maas-api-tls\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.745524 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.745452    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-metrics-certs\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:52.804700 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:52.804669    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkq87\" (UniqueName: \"kubernetes.io/projected/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-kube-api-access-jkq87\") pod \"maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:53.082216 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:53.082180    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:53.465905 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:53.465874    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"]
Nov 02 01:59:53.466528 ip-10-0-134-183 kubenswrapper[4287]: W1102 01:59:53.466504    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df408c8_a8fa_4c65_b307_1fc0c6c43c87.slice/crio-2ba0ee8b2d5b71601e800bfb1ab14dff4ca84ec05922bb20a1a4e401f925224a WatchSource:0}: Error finding container 2ba0ee8b2d5b71601e800bfb1ab14dff4ca84ec05922bb20a1a4e401f925224a: Status 404 returned error can't find the container with id 2ba0ee8b2d5b71601e800bfb1ab14dff4ca84ec05922bb20a1a4e401f925224a
Nov 02 01:59:53.679785 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:53.679748    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq" event={"ID":"5df408c8-a8fa-4c65-b307-1fc0c6c43c87","Type":"ContainerStarted","Data":"2ba0ee8b2d5b71601e800bfb1ab14dff4ca84ec05922bb20a1a4e401f925224a"}
Nov 02 01:59:54.689346 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:54.689243    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq" event={"ID":"5df408c8-a8fa-4c65-b307-1fc0c6c43c87","Type":"ContainerStarted","Data":"e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901"}
Nov 02 01:59:54.689791 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:54.689428    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 01:59:54.812666 ip-10-0-134-183 kubenswrapper[4287]: I1102 01:59:54.812573    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq" podStartSLOduration=1.9661490019999999 podStartE2EDuration="2.812549341s" totalImagesPullingTime="846.400339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:59:54.808738568 +0000 UTC m=+2102.682379621" watchObservedRunningTime="2026-11-02 01:59:54.812549341 +0000 UTC m=+2102.686190393"
Nov 02 02:00:00.220513 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.220473    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"]
Nov 02 02:00:00.224803 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.224776    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"]
Nov 02 02:00:00.224931 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.224894    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:00.228574 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.228548    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:00.249124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.249085    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"]
Nov 02 02:00:00.272977 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.272941    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:00.317701 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.317667    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"]
Nov 02 02:00:00.323724 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.323635    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"]
Nov 02 02:00:00.340934 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.340881    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"]
Nov 02 02:00:00.361962 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.361930    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\""
Nov 02 02:00:00.366964 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.366920    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73fae3f-e816-4a5c-9a51-dbb216f7b123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-9kjgx\" (UID: \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:00.367165 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.366977    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtxv\" (UniqueName: \"kubernetes.io/projected/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-kube-api-access-ndtxv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh\" (UID: \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:00.367245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.367181    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dmm\" (UniqueName: \"kubernetes.io/projected/d73fae3f-e816-4a5c-9a51-dbb216f7b123-kube-api-access-z2dmm\") pod \"maas-api-key-cleanup-29893080-9kjgx\" (UID: \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:00.367305 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.367284    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lgw\" (UniqueName: \"kubernetes.io/projected/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-kube-api-access-s9lgw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s\" (UID: \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:00.367389 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.367372    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh\" (UID: \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:00.367509 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.367494    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s\" (UID: \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:00.468803 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.468766    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73fae3f-e816-4a5c-9a51-dbb216f7b123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-9kjgx\" (UID: \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:00.468803 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.468806    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtxv\" (UniqueName: \"kubernetes.io/projected/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-kube-api-access-ndtxv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh\" (UID: \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:00.469047 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.468871    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dmm\" (UniqueName: \"kubernetes.io/projected/d73fae3f-e816-4a5c-9a51-dbb216f7b123-kube-api-access-z2dmm\") pod \"maas-api-key-cleanup-29893080-9kjgx\" (UID: \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:00.469047 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.468903    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lgw\" (UniqueName: \"kubernetes.io/projected/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-kube-api-access-s9lgw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s\" (UID: \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:00.469047 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.468977    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh\" (UID: \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:00.469198 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.469092    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s\" (UID: \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:00.469682 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.469590    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73fae3f-e816-4a5c-9a51-dbb216f7b123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893080-9kjgx\" (UID: \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:00.469805 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.469789    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh\" (UID: \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:00.469847 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.469832    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s\" (UID: \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:00.529602 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.529562    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lgw\" (UniqueName: \"kubernetes.io/projected/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-kube-api-access-s9lgw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s\" (UID: \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:00.529602 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.529588    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtxv\" (UniqueName: \"kubernetes.io/projected/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-kube-api-access-ndtxv\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh\" (UID: \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:00.529825 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.529588    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dmm\" (UniqueName: \"kubernetes.io/projected/d73fae3f-e816-4a5c-9a51-dbb216f7b123-kube-api-access-z2dmm\") pod \"maas-api-key-cleanup-29893080-9kjgx\" (UID: \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:00.541574 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.541542    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:00.562018 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.561987    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:00.592139 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.592104    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:00.705374 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.704192    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 02:00:00.924083 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:00:00.924050    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb40c3dd_a1ee_4cde_956b_1c68a3aa5b6e.slice/crio-8d21e3606ae0c8fd03ef3aa40cf9aa2ef86d049f0de7670289839c8b1a3c11b0 WatchSource:0}: Error finding container 8d21e3606ae0c8fd03ef3aa40cf9aa2ef86d049f0de7670289839c8b1a3c11b0: Status 404 returned error can't find the container with id 8d21e3606ae0c8fd03ef3aa40cf9aa2ef86d049f0de7670289839c8b1a3c11b0
Nov 02 02:00:00.924196 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.924138    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"]
Nov 02 02:00:00.975186 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:00.975148    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"]
Nov 02 02:00:00.975624 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:00:00.975580    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a387fa_d0a0_4f6c_bb5a_41cb5bd35d44.slice/crio-e2192aa3e01a6105895376830b8f654231e8694907e2f825118834471105942c WatchSource:0}: Error finding container e2192aa3e01a6105895376830b8f654231e8694907e2f825118834471105942c: Status 404 returned error can't find the container with id e2192aa3e01a6105895376830b8f654231e8694907e2f825118834471105942c
Nov 02 02:00:01.027504 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:00:01.027458    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73fae3f_e816_4a5c_9a51_dbb216f7b123.slice/crio-3d17ff26eea82fb3ebaafe0d656989d2dbc6a98ee97f64dc16d067a7fc0f69b8 WatchSource:0}: Error finding container 3d17ff26eea82fb3ebaafe0d656989d2dbc6a98ee97f64dc16d067a7fc0f69b8: Status 404 returned error can't find the container with id 3d17ff26eea82fb3ebaafe0d656989d2dbc6a98ee97f64dc16d067a7fc0f69b8
Nov 02 02:00:01.027792 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:01.027767    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"]
Nov 02 02:00:01.741268 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:01.741230    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" event={"ID":"d73fae3f-e816-4a5c-9a51-dbb216f7b123","Type":"ContainerStarted","Data":"3d17ff26eea82fb3ebaafe0d656989d2dbc6a98ee97f64dc16d067a7fc0f69b8"}
Nov 02 02:00:01.742513 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:01.742486    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" event={"ID":"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44","Type":"ContainerStarted","Data":"e2192aa3e01a6105895376830b8f654231e8694907e2f825118834471105942c"}
Nov 02 02:00:01.743808 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:01.743782    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" event={"ID":"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e","Type":"ContainerStarted","Data":"8d21e3606ae0c8fd03ef3aa40cf9aa2ef86d049f0de7670289839c8b1a3c11b0"}
Nov 02 02:00:07.787910 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:07.787871    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" event={"ID":"d73fae3f-e816-4a5c-9a51-dbb216f7b123","Type":"ContainerStarted","Data":"dc7631b3ce60cb40d768111a77878e27848c463ccda9f81ae4de6758f09b949a"}
Nov 02 02:00:07.789561 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:07.789535    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" event={"ID":"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e","Type":"ContainerStarted","Data":"c15aa1847fa91bdb755568d1f7a23c65c70466c1fb32d17adbf0f8a7e792100a"}
Nov 02 02:00:07.906827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:07.906763    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" podStartSLOduration=1.617277909 podStartE2EDuration="7.906747694s" totalImagesPullingTime="6.289469785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:07.902319061 +0000 UTC m=+2115.775960115" watchObservedRunningTime="2026-11-02 02:00:07.906747694 +0000 UTC m=+2115.780388750"
Nov 02 02:00:08.018007 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.017953    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" podStartSLOduration=2.08029271 podStartE2EDuration="8.0179382s" totalImagesPullingTime="5.93764549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:08.016007421 +0000 UTC m=+2115.889648473" watchObservedRunningTime="2026-11-02 02:00:08.0179382 +0000 UTC m=+2115.891579251"
Nov 02 02:00:08.797347 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.797316    4287 generic.go:356] "Generic (PLEG): container finished" podID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerID="c15aa1847fa91bdb755568d1f7a23c65c70466c1fb32d17adbf0f8a7e792100a" exitCode=60
Nov 02 02:00:08.797872 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.797410    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" event={"ID":"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e","Type":"ContainerDied","Data":"c15aa1847fa91bdb755568d1f7a23c65c70466c1fb32d17adbf0f8a7e792100a"}
Nov 02 02:00:08.797872 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.797737    4287 scope.go:122] "RemoveContainer" containerID="c15aa1847fa91bdb755568d1f7a23c65c70466c1fb32d17adbf0f8a7e792100a"
Nov 02 02:00:08.799503 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.799478    4287 generic.go:356] "Generic (PLEG): container finished" podID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerID="dc7631b3ce60cb40d768111a77878e27848c463ccda9f81ae4de6758f09b949a" exitCode=60
Nov 02 02:00:08.799598 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.799565    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" event={"ID":"d73fae3f-e816-4a5c-9a51-dbb216f7b123","Type":"ContainerDied","Data":"dc7631b3ce60cb40d768111a77878e27848c463ccda9f81ae4de6758f09b949a"}
Nov 02 02:00:08.799940 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.799918    4287 scope.go:122] "RemoveContainer" containerID="dc7631b3ce60cb40d768111a77878e27848c463ccda9f81ae4de6758f09b949a"
Nov 02 02:00:08.801621 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.801578    4287 generic.go:356] "Generic (PLEG): container finished" podID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerID="dbc42364ab21db9d33ba723afadcdd9fb1dafd3d4a1cce0d5c346ffac5c0d301" exitCode=60
Nov 02 02:00:08.801710 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.801657    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" event={"ID":"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44","Type":"ContainerDied","Data":"dbc42364ab21db9d33ba723afadcdd9fb1dafd3d4a1cce0d5c346ffac5c0d301"}
Nov 02 02:00:08.801895 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:08.801881    4287 scope.go:122] "RemoveContainer" containerID="dbc42364ab21db9d33ba723afadcdd9fb1dafd3d4a1cce0d5c346ffac5c0d301"
Nov 02 02:00:09.811719 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.811686    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" event={"ID":"d73fae3f-e816-4a5c-9a51-dbb216f7b123","Type":"ContainerStarted","Data":"587cef4c12a33fc2049d8df7f7213e7f2084e0a5972b0088c336a7790c0fa56d"}
Nov 02 02:00:09.813804 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.813775    4287 generic.go:356] "Generic (PLEG): container finished" podID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerID="69e5205b439cd8dc943637c86a374545e1615e68b36f5619b8c32b8006e4e30b" exitCode=60
Nov 02 02:00:09.813920 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.813858    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" event={"ID":"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44","Type":"ContainerDied","Data":"69e5205b439cd8dc943637c86a374545e1615e68b36f5619b8c32b8006e4e30b"}
Nov 02 02:00:09.813920 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.813906    4287 scope.go:122] "RemoveContainer" containerID="dbc42364ab21db9d33ba723afadcdd9fb1dafd3d4a1cce0d5c346ffac5c0d301"
Nov 02 02:00:09.814130 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.814111    4287 scope.go:122] "RemoveContainer" containerID="69e5205b439cd8dc943637c86a374545e1615e68b36f5619b8c32b8006e4e30b"
Nov 02 02:00:09.814453 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:09.814428    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s_odh-ai-gateway-infra(50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44"
Nov 02 02:00:09.815903 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.815882    4287 generic.go:356] "Generic (PLEG): container finished" podID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerID="14b276c14bd942c9ae4f06408ad3578b23faa9d9e0856834c345c1c6b5bf38de" exitCode=60
Nov 02 02:00:09.816006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.815946    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" event={"ID":"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e","Type":"ContainerDied","Data":"14b276c14bd942c9ae4f06408ad3578b23faa9d9e0856834c345c1c6b5bf38de"}
Nov 02 02:00:09.816233 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.816218    4287 scope.go:122] "RemoveContainer" containerID="14b276c14bd942c9ae4f06408ad3578b23faa9d9e0856834c345c1c6b5bf38de"
Nov 02 02:00:09.816539 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:09.816518    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh_odh-ai-gateway-infra(bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e"
Nov 02 02:00:09.826813 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:09.826782    4287 scope.go:122] "RemoveContainer" containerID="c15aa1847fa91bdb755568d1f7a23c65c70466c1fb32d17adbf0f8a7e792100a"
Nov 02 02:00:10.827187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:10.827155    4287 generic.go:356] "Generic (PLEG): container finished" podID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerID="587cef4c12a33fc2049d8df7f7213e7f2084e0a5972b0088c336a7790c0fa56d" exitCode=60
Nov 02 02:00:10.827694 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:10.827237    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" event={"ID":"d73fae3f-e816-4a5c-9a51-dbb216f7b123","Type":"ContainerDied","Data":"587cef4c12a33fc2049d8df7f7213e7f2084e0a5972b0088c336a7790c0fa56d"}
Nov 02 02:00:10.827694 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:10.827309    4287 scope.go:122] "RemoveContainer" containerID="dc7631b3ce60cb40d768111a77878e27848c463ccda9f81ae4de6758f09b949a"
Nov 02 02:00:10.827694 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:10.827564    4287 scope.go:122] "RemoveContainer" containerID="587cef4c12a33fc2049d8df7f7213e7f2084e0a5972b0088c336a7790c0fa56d"
Nov 02 02:00:10.827966 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:10.827944    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-9kjgx_odh-ai-gateway-infra(d73fae3f-e816-4a5c-9a51-dbb216f7b123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123"
Nov 02 02:00:10.829267 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:10.829251    4287 scope.go:122] "RemoveContainer" containerID="69e5205b439cd8dc943637c86a374545e1615e68b36f5619b8c32b8006e4e30b"
Nov 02 02:00:10.829569 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:10.829547    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s_odh-ai-gateway-infra(50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44"
Nov 02 02:00:10.830896 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:10.830880    4287 scope.go:122] "RemoveContainer" containerID="14b276c14bd942c9ae4f06408ad3578b23faa9d9e0856834c345c1c6b5bf38de"
Nov 02 02:00:10.831127 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:10.831110    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh_odh-ai-gateway-infra(bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e"
Nov 02 02:00:11.838558 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:11.838526    4287 scope.go:122] "RemoveContainer" containerID="587cef4c12a33fc2049d8df7f7213e7f2084e0a5972b0088c336a7790c0fa56d"
Nov 02 02:00:11.838966 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:11.838864    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-9kjgx_odh-ai-gateway-infra(d73fae3f-e816-4a5c-9a51-dbb216f7b123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123"
Nov 02 02:00:15.537748 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:15.537714    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"]
Nov 02 02:00:15.540309 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:15.537941    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy" containerID="cri-o://f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7" gracePeriod=30
Nov 02 02:00:16.041642 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:16.041572    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:00:20.803737 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.803671    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 02:00:20.901076 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901042    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-certs\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901268 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901088    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-token\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-token\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901268 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901158    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-data\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901268 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901207    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-envoy\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901268 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901236    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-credential-socket\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901268 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901262    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-podinfo\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901539 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901373    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-socket\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901539 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901425    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-certs" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:20.901539 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901501    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-kube-api-access-z58gz\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-kube-api-access-z58gz\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901782 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901551    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istiod-ca-cert\") pod \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\" (UID: \"a85f82d1-b482-4ae5-b9b8-e335e871f8e2\") "
Nov 02 02:00:20.901782 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901573    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-credential-socket" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:20.901782 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901754    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-socket" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:20.901994 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.901968    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-data" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:20.902111 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.902093    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istiod-ca-cert" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:20.902487 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.902458    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.902487 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.902485    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.902882 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.902501    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.902882 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.902517    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.902882 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.902531    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:20.903863 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.903834    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-token" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:20.903960 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.903860    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-envoy" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:00:20.904274 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.904255    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-podinfo" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:00:20.904330 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.904306    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-kube-api-access-z58gz" pod "a85f82d1-b482-4ae5-b9b8-e335e871f8e2" (UID: "a85f82d1-b482-4ae5-b9b8-e335e871f8e2"). InnerVolumeSpecName "kube-api-access-z58gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:20.926235 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.926206    4287 generic.go:356] "Generic (PLEG): container finished" podID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerID="f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7" exitCode=0
Nov 02 02:00:20.926382 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.926266    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"
Nov 02 02:00:20.926382 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.926289    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" event={"ID":"a85f82d1-b482-4ae5-b9b8-e335e871f8e2","Type":"ContainerDied","Data":"f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7"}
Nov 02 02:00:20.926382 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.926357    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n" event={"ID":"a85f82d1-b482-4ae5-b9b8-e335e871f8e2","Type":"ContainerDied","Data":"e5b093b84bae4ddd828b67c36080df05d9d4179c628cfd27ad8fda3c926deee1"}
Nov 02 02:00:20.926382 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.926381    4287 scope.go:122] "RemoveContainer" containerID="f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7"
Nov 02 02:00:20.938689 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.938667    4287 scope.go:122] "RemoveContainer" containerID="f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7"
Nov 02 02:00:20.938974 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:20.938955    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7\": container with ID starting with f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7 not found: ID does not exist" containerID="f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7"
Nov 02 02:00:20.939022 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:20.938985    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7"} err="failed to get container status \"f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7\": rpc error: code = NotFound desc = could not find container \"f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7\": container with ID starting with f426011b606d283f2e42f8efee307f12ff772d890b2c6c6f95b9dd14d493c5c7 not found: ID does not exist"
Nov 02 02:00:21.004529 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:21.004492    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z58gz\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-kube-api-access-z58gz\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.004800 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:21.004533    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.004800 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:21.004551    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.004800 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:21.004579    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a85f82d1-b482-4ae5-b9b8-e335e871f8e2-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:21.093597 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:21.093511    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"]
Nov 02 02:00:21.101114 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:21.101074    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-sw67n"]
Nov 02 02:00:22.243722 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:22.243691    4287 scope.go:122] "RemoveContainer" containerID="587cef4c12a33fc2049d8df7f7213e7f2084e0a5972b0088c336a7790c0fa56d"
Nov 02 02:00:22.250741 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:22.250716    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" path="/var/lib/kubelet/pods/a85f82d1-b482-4ae5-b9b8-e335e871f8e2/volumes"
Nov 02 02:00:22.942121 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:22.942036    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" event={"ID":"d73fae3f-e816-4a5c-9a51-dbb216f7b123","Type":"ContainerStarted","Data":"f91a1df7cfcc3d4e5801dbb60c86ac513ec5b4859ea3c5119e269768db007dbe"}
Nov 02 02:00:23.230869 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:23.230787    4287 scope.go:122] "RemoveContainer" containerID="69e5205b439cd8dc943637c86a374545e1615e68b36f5619b8c32b8006e4e30b"
Nov 02 02:00:23.231091 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:23.231073    4287 scope.go:122] "RemoveContainer" containerID="14b276c14bd942c9ae4f06408ad3578b23faa9d9e0856834c345c1c6b5bf38de"
Nov 02 02:00:23.950752 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:23.950656    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" event={"ID":"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e","Type":"ContainerStarted","Data":"6d22020e88b86c3d7fbbfc6952ad2f0722f6f555d4f1a7ac1e42b5265b7782ae"}
Nov 02 02:00:23.952770 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:23.952746    4287 generic.go:356] "Generic (PLEG): container finished" podID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerID="f91a1df7cfcc3d4e5801dbb60c86ac513ec5b4859ea3c5119e269768db007dbe" exitCode=60
Nov 02 02:00:23.952901 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:23.952817    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" event={"ID":"d73fae3f-e816-4a5c-9a51-dbb216f7b123","Type":"ContainerDied","Data":"f91a1df7cfcc3d4e5801dbb60c86ac513ec5b4859ea3c5119e269768db007dbe"}
Nov 02 02:00:23.952901 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:23.952867    4287 scope.go:122] "RemoveContainer" containerID="587cef4c12a33fc2049d8df7f7213e7f2084e0a5972b0088c336a7790c0fa56d"
Nov 02 02:00:23.953310 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:23.953286    4287 scope.go:122] "RemoveContainer" containerID="f91a1df7cfcc3d4e5801dbb60c86ac513ec5b4859ea3c5119e269768db007dbe"
Nov 02 02:00:23.953671 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:23.953648    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893080-9kjgx_odh-ai-gateway-infra(d73fae3f-e816-4a5c-9a51-dbb216f7b123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123"
Nov 02 02:00:23.954889 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:23.954869    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" event={"ID":"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44","Type":"ContainerStarted","Data":"2f1f430536326b9661f9e592ffc4050db1b3074b4d5aaa8a974fd98fb3fc5430"}
Nov 02 02:00:24.067851 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.067821    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"]
Nov 02 02:00:24.184061 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.183996    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" podStartSLOduration=17.333659209 podStartE2EDuration="24.183981477s" totalImagesPullingTime="6.850322268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:24.180361439 +0000 UTC m=+2132.054002491" watchObservedRunningTime="2026-11-02 02:00:24.183981477 +0000 UTC m=+2132.057622548"
Nov 02 02:00:24.965796 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.965763    4287 generic.go:356] "Generic (PLEG): container finished" podID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerID="2f1f430536326b9661f9e592ffc4050db1b3074b4d5aaa8a974fd98fb3fc5430" exitCode=60
Nov 02 02:00:24.966202 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.965837    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" event={"ID":"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44","Type":"ContainerDied","Data":"2f1f430536326b9661f9e592ffc4050db1b3074b4d5aaa8a974fd98fb3fc5430"}
Nov 02 02:00:24.966202 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.965894    4287 scope.go:122] "RemoveContainer" containerID="69e5205b439cd8dc943637c86a374545e1615e68b36f5619b8c32b8006e4e30b"
Nov 02 02:00:24.966342 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.966317    4287 scope.go:122] "RemoveContainer" containerID="2f1f430536326b9661f9e592ffc4050db1b3074b4d5aaa8a974fd98fb3fc5430"
Nov 02 02:00:24.966698 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:24.966671    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s_odh-ai-gateway-infra(50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44"
Nov 02 02:00:24.967978 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.967959    4287 generic.go:356] "Generic (PLEG): container finished" podID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerID="6d22020e88b86c3d7fbbfc6952ad2f0722f6f555d4f1a7ac1e42b5265b7782ae" exitCode=60
Nov 02 02:00:24.968052 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.968037    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" event={"ID":"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e","Type":"ContainerDied","Data":"6d22020e88b86c3d7fbbfc6952ad2f0722f6f555d4f1a7ac1e42b5265b7782ae"}
Nov 02 02:00:24.968377 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.968363    4287 scope.go:122] "RemoveContainer" containerID="6d22020e88b86c3d7fbbfc6952ad2f0722f6f555d4f1a7ac1e42b5265b7782ae"
Nov 02 02:00:24.968601 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:24.968585    4287 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh_odh-ai-gateway-infra(bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e"
Nov 02 02:00:24.980547 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:24.980522    4287 scope.go:122] "RemoveContainer" containerID="14b276c14bd942c9ae4f06408ad3578b23faa9d9e0856834c345c1c6b5bf38de"
Nov 02 02:00:25.079113 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.079082    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"]
Nov 02 02:00:25.133901 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.133877    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:25.196971 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.196934    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"]
Nov 02 02:00:25.261931 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.261907    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73fae3f-e816-4a5c-9a51-dbb216f7b123-kube-api-access-z2dmm\" (UniqueName: \"kubernetes.io/projected/d73fae3f-e816-4a5c-9a51-dbb216f7b123-kube-api-access-z2dmm\") pod \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\" (UID: \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\") "
Nov 02 02:00:25.262083 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.262063    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73fae3f-e816-4a5c-9a51-dbb216f7b123-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73fae3f-e816-4a5c-9a51-dbb216f7b123-maas-api-service-ca\") pod \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\" (UID: \"d73fae3f-e816-4a5c-9a51-dbb216f7b123\") "
Nov 02 02:00:25.262471 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.262443    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73fae3f-e816-4a5c-9a51-dbb216f7b123-maas-api-service-ca" pod "d73fae3f-e816-4a5c-9a51-dbb216f7b123" (UID: "d73fae3f-e816-4a5c-9a51-dbb216f7b123"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:25.262749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.262559    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73fae3f-e816-4a5c-9a51-dbb216f7b123-maas-api-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:25.264149 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.264127    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73fae3f-e816-4a5c-9a51-dbb216f7b123-kube-api-access-z2dmm" pod "d73fae3f-e816-4a5c-9a51-dbb216f7b123" (UID: "d73fae3f-e816-4a5c-9a51-dbb216f7b123"). InnerVolumeSpecName "kube-api-access-z2dmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:25.364671 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:25.364643    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dmm\" (UniqueName: \"kubernetes.io/projected/d73fae3f-e816-4a5c-9a51-dbb216f7b123-kube-api-access-z2dmm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:26.067630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.056660    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx" event={"ID":"d73fae3f-e816-4a5c-9a51-dbb216f7b123","Type":"ContainerDied","Data":"3d17ff26eea82fb3ebaafe0d656989d2dbc6a98ee97f64dc16d067a7fc0f69b8"}
Nov 02 02:00:26.067630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.056736    4287 scope.go:122] "RemoveContainer" containerID="f91a1df7cfcc3d4e5801dbb60c86ac513ec5b4859ea3c5119e269768db007dbe"
Nov 02 02:00:26.067630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.056985    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"
Nov 02 02:00:26.404408 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.404360    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:26.427485 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.427459    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:26.431963 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.431940    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-kube-api-access-s9lgw\" (UniqueName: \"kubernetes.io/projected/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-kube-api-access-s9lgw\") pod \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\" (UID: \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\") "
Nov 02 02:00:26.432128 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.432114    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-maas-api-service-ca\") pod \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\" (UID: \"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44\") "
Nov 02 02:00:26.432493 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.432469    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-maas-api-service-ca" pod "50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" (UID: "50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:26.432656 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.432641    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-maas-api-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:26.434176 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.434154    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-kube-api-access-s9lgw" pod "50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" (UID: "50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44"). InnerVolumeSpecName "kube-api-access-s9lgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:26.504295 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.504186    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"]
Nov 02 02:00:26.507636 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.507600    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893080-9kjgx"]
Nov 02 02:00:26.533646 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.533593    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-kube-api-access-ndtxv\" (UniqueName: \"kubernetes.io/projected/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-kube-api-access-ndtxv\") pod \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\" (UID: \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\") "
Nov 02 02:00:26.533785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.533694    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-maas-api-service-ca\") pod \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\" (UID: \"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e\") "
Nov 02 02:00:26.534043 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.534029    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9lgw\" (UniqueName: \"kubernetes.io/projected/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44-kube-api-access-s9lgw\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:26.534100 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.534042    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-maas-api-service-ca" pod "bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" (UID: "bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:00:26.535922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.535901    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-kube-api-access-ndtxv" pod "bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" (UID: "bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e"). InnerVolumeSpecName "kube-api-access-ndtxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:00:26.635333 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.635256    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-maas-api-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:26.635333 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:26.635291    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtxv\" (UniqueName: \"kubernetes.io/projected/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e-kube-api-access-ndtxv\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:00:27.078281 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.078251    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"
Nov 02 02:00:27.078791 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.078287    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh" event={"ID":"bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e","Type":"ContainerDied","Data":"8d21e3606ae0c8fd03ef3aa40cf9aa2ef86d049f0de7670289839c8b1a3c11b0"}
Nov 02 02:00:27.078791 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.078354    4287 scope.go:122] "RemoveContainer" containerID="6d22020e88b86c3d7fbbfc6952ad2f0722f6f555d4f1a7ac1e42b5265b7782ae"
Nov 02 02:00:27.081318 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.081290    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s" event={"ID":"50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44","Type":"ContainerDied","Data":"e2192aa3e01a6105895376830b8f654231e8694907e2f825118834471105942c"}
Nov 02 02:00:27.081439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.081303    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"
Nov 02 02:00:27.089038 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.088870    4287 scope.go:122] "RemoveContainer" containerID="2f1f430536326b9661f9e592ffc4050db1b3074b4d5aaa8a974fd98fb3fc5430"
Nov 02 02:00:27.243857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.243830    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"]
Nov 02 02:00:27.292862 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.292833    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ef5a72-29893080-nvvzh"]
Nov 02 02:00:27.357767 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.357598    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"]
Nov 02 02:00:27.361321 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:27.361297    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-218802-29893080-t952s"]
Nov 02 02:00:28.238237 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:28.238202    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" path="/var/lib/kubelet/pods/50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44/volumes"
Nov 02 02:00:28.238717 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:28.238698    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" path="/var/lib/kubelet/pods/bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e/volumes"
Nov 02 02:00:28.239150 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:28.239136    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" path="/var/lib/kubelet/pods/d73fae3f-e816-4a5c-9a51-dbb216f7b123/volumes"
Nov 02 02:00:46.039464 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:46.039420    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:00:50.642642 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.642594    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"]
Nov 02 02:00:50.643144 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643132    4287 cpu_manager.go:414] "containerMap: removing container" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:00:50.643187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643146    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:00:50.643187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643159    4287 cpu_manager.go:414] "containerMap: removing container" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:00:50.643187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643163    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:00:50.643187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643174    4287 cpu_manager.go:414] "containerMap: removing container" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:00:50.643187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643179    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:00:50.643187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643187    4287 cpu_manager.go:414] "containerMap: removing container" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643193    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643208    4287 cpu_manager.go:414] "containerMap: removing container" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643212    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643221    4287 cpu_manager.go:414] "containerMap: removing container" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643225    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643237    4287 cpu_manager.go:414] "containerMap: removing container" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643241    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643253    4287 cpu_manager.go:414] "containerMap: removing container" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643258    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643327    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85f82d1-b482-4ae5-b9b8-e335e871f8e2" containerName="istio-proxy"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643336    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643343    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643349    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643355    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643362    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643366    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643370    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:00:50.643378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.643377    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:00:50.646769 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.646750    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.742036 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.741998    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.742036 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.742043    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-openshift-service-ca\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.742403 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.742371    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnw2j\" (UniqueName: \"kubernetes.io/projected/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-kube-api-access-vnw2j\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.742486 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.742470    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-metrics-certs\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.750434 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.750411    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-269235-serving-cert\""
Nov 02 02:00:50.754490 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.754468    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"]
Nov 02 02:00:50.843878 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.843847    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-openshift-service-ca\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.844057 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.844023    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnw2j\" (UniqueName: \"kubernetes.io/projected/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-kube-api-access-vnw2j\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.844111 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.844059    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-metrics-certs\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.844111 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.844101    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.844217 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:50.844207    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-269235-serving-cert: secret "maas-api-e2e-api-a-269235-serving-cert" not found
Nov 02 02:00:50.844284 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:00:50.844272    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls podName:4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1 nodeName:}" failed. No retries permitted until 2026-11-02 02:00:51.344254314 +0000 UTC m=+2159.217895357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls") pod "maas-api-e2e-api-a-269235-787486f758-hqkqz" (UID: "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1") : secret "maas-api-e2e-api-a-269235-serving-cert" not found
Nov 02 02:00:50.844637 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.844582    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-openshift-service-ca\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.846955 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.846934    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-metrics-certs\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:50.907322 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:50.907246    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnw2j\" (UniqueName: \"kubernetes.io/projected/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-kube-api-access-vnw2j\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:51.358923 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:51.358875    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:51.361456 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:51.361429    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls\") pod \"maas-api-e2e-api-a-269235-787486f758-hqkqz\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:51.559839 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:51.559800    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:51.942415 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:51.942382    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"]
Nov 02 02:00:51.943110 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:00:51.943084    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9088ae_b6ab_4976_9fa0_c77a3bec3ec1.slice/crio-f48624b9258e992b6c395b26ce2b0bb8306a422bf75f0902920c20974c88b3d6 WatchSource:0}: Error finding container f48624b9258e992b6c395b26ce2b0bb8306a422bf75f0902920c20974c88b3d6: Status 404 returned error can't find the container with id f48624b9258e992b6c395b26ce2b0bb8306a422bf75f0902920c20974c88b3d6
Nov 02 02:00:52.263519 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:52.263479    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz" event={"ID":"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1","Type":"ContainerStarted","Data":"f48624b9258e992b6c395b26ce2b0bb8306a422bf75f0902920c20974c88b3d6"}
Nov 02 02:00:53.271505 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:53.271468    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz" event={"ID":"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1","Type":"ContainerStarted","Data":"a235768bb8224810b8fd7bac90fdd3dd501fead529bd06996f038f249440baee"}
Nov 02 02:00:53.272004 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:53.271639    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:00:53.390426 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:53.390373    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz" podStartSLOduration=2.5387625480000002 podStartE2EDuration="3.390360062s" totalImagesPullingTime="851.597514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:53.387890338 +0000 UTC m=+2161.261531391" watchObservedRunningTime="2026-11-02 02:00:53.390360062 +0000 UTC m=+2161.264001113"
Nov 02 02:00:59.283938 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:00:59.283904    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:01:01.073090 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:01.073058    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"]
Nov 02 02:01:01.073492 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:01.073288    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy" containerID="cri-o://4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488" gracePeriod=30
Nov 02 02:01:05.264013 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.263980    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"]
Nov 02 02:01:05.264513 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.264500    4287 cpu_manager.go:414] "containerMap: removing container" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:01:05.264558 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.264514    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a387fa-d0a0-4f6c-bb5a-41cb5bd35d44" containerName="cleanup"
Nov 02 02:01:05.264558 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.264546    4287 cpu_manager.go:414] "containerMap: removing container" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:01:05.264558 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.264551    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb40c3dd-a1ee-4cde-956b-1c68a3aa5b6e" containerName="cleanup"
Nov 02 02:01:05.264698 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.264656    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73fae3f-e816-4a5c-9a51-dbb216f7b123" containerName="cleanup"
Nov 02 02:01:05.268169 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.268152    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.322980 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.322938    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-69c15d-serving-cert\""
Nov 02 02:01:05.360756 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.360728    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-metrics-certs\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.360959 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.360891    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.361052 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.361027    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708ed890-4992-4dab-9878-b000a9c3de79-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.361188 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.361172    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqf5\" (UniqueName: \"kubernetes.io/projected/708ed890-4992-4dab-9878-b000a9c3de79-kube-api-access-vqqf5\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.376028 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.376001    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"]
Nov 02 02:01:05.461971 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.461933    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708ed890-4992-4dab-9878-b000a9c3de79-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.462136 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.462061    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqf5\" (UniqueName: \"kubernetes.io/projected/708ed890-4992-4dab-9878-b000a9c3de79-kube-api-access-vqqf5\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.462136 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.462093    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-metrics-certs\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.462211 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.462194    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.462363 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:01:05.462347    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-serving-cert: secret "maas-api-e2e-inf-a-69c15d-serving-cert" not found
Nov 02 02:01:05.462422 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:01:05.462413    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls podName:708ed890-4992-4dab-9878-b000a9c3de79 nodeName:}" failed. No retries permitted until 2026-11-02 02:01:05.962396032 +0000 UTC m=+2173.836037067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls") pod "maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs" (UID: "708ed890-4992-4dab-9878-b000a9c3de79") : secret "maas-api-e2e-inf-a-69c15d-serving-cert" not found
Nov 02 02:01:05.462682 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.462661    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708ed890-4992-4dab-9878-b000a9c3de79-openshift-service-ca\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.465112 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.465089    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-metrics-certs\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.524904 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.524807    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqf5\" (UniqueName: \"kubernetes.io/projected/708ed890-4992-4dab-9878-b000a9c3de79-kube-api-access-vqqf5\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.968454 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:05.968365    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:05.968683 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:01:05.968540    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-serving-cert: secret "maas-api-e2e-inf-a-69c15d-serving-cert" not found
Nov 02 02:01:05.968683 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:01:05.968658    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls podName:708ed890-4992-4dab-9878-b000a9c3de79 nodeName:}" failed. No retries permitted until 2026-11-02 02:01:06.968599265 +0000 UTC m=+2174.842240312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls") pod "maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs" (UID: "708ed890-4992-4dab-9878-b000a9c3de79") : secret "maas-api-e2e-inf-a-69c15d-serving-cert" not found
Nov 02 02:01:06.356674 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.356648    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 02:01:06.359386 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.359356    4287 generic.go:356] "Generic (PLEG): container finished" podID="e162987e-acf2-4901-8631-822c903e32d8" containerID="4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488" exitCode=0
Nov 02 02:01:06.359529 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.359402    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"
Nov 02 02:01:06.359529 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.359411    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" event={"ID":"e162987e-acf2-4901-8631-822c903e32d8","Type":"ContainerDied","Data":"4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488"}
Nov 02 02:01:06.359529 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.359464    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l" event={"ID":"e162987e-acf2-4901-8631-822c903e32d8","Type":"ContainerDied","Data":"1f74fd66b6747416397fea00a0677ef30338b2461440bb81f788b04737c5bfd1"}
Nov 02 02:01:06.359529 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.359494    4287 scope.go:122] "RemoveContainer" containerID="4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488"
Nov 02 02:01:06.369411 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.369391    4287 scope.go:122] "RemoveContainer" containerID="4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488"
Nov 02 02:01:06.369721 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:01:06.369701    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488\": container with ID starting with 4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488 not found: ID does not exist" containerID="4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488"
Nov 02 02:01:06.369775 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.369730    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488"} err="failed to get container status \"4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488\": rpc error: code = NotFound desc = could not find container \"4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488\": container with ID starting with 4a5fc9a9b0cd922e38215f3b5ef7410db2b55b9ba11d9c0f2b3666dcbc4de488 not found: ID does not exist"
Nov 02 02:01:06.373141 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373117    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-credential-socket\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373289 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373278    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-data\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373333 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373300    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-envoy\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373391 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373342    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e162987e-acf2-4901-8631-822c903e32d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162987e-acf2-4901-8631-822c903e32d8-istiod-ca-cert\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373391 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373350    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-credential-socket" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:06.373391 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373373    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e162987e-acf2-4901-8631-822c903e32d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162987e-acf2-4901-8631-822c903e32d8-istio-podinfo\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373547 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373465    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-socket\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373547 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373519    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-kube-api-access-ftshq\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-kube-api-access-ftshq\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373665 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373580    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-certs\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373665 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373641    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-data" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:06.373777 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373759    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-istio-token\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-istio-token\") pod \"e162987e-acf2-4901-8631-822c903e32d8\" (UID: \"e162987e-acf2-4901-8631-822c903e32d8\") "
Nov 02 02:01:06.373839 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373778    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-socket" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:06.373903 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373789    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e162987e-acf2-4901-8631-822c903e32d8-istiod-ca-cert" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:06.373903 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.373868    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-certs" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:06.374583 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.374556    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e162987e-acf2-4901-8631-822c903e32d8-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.374701 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.374587    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.374701 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.374636    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.374701 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.374651    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.374701 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.374665    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.375720 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.375698    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e162987e-acf2-4901-8631-822c903e32d8-istio-podinfo" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:01:06.375720 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.375707    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-envoy" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:01:06.375867 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.375854    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-istio-token" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:06.375996 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.375978    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-kube-api-access-ftshq" pod "e162987e-acf2-4901-8631-822c903e32d8" (UID: "e162987e-acf2-4901-8631-822c903e32d8"). InnerVolumeSpecName "kube-api-access-ftshq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:06.476289 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.476261    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e162987e-acf2-4901-8631-822c903e32d8-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.476289 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.476289    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e162987e-acf2-4901-8631-822c903e32d8-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.476450 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.476299    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftshq\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-kube-api-access-ftshq\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.476450 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.476310    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e162987e-acf2-4901-8631-822c903e32d8-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:06.787468 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.787429    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"]
Nov 02 02:01:06.790918 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.790894    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-h9x6l"]
Nov 02 02:01:06.982427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.982333    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:06.985025 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:06.985004    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls\") pod \"maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:07.082052 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:07.082012    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:07.464669 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:07.464640    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"]
Nov 02 02:01:07.465624 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:01:07.465573    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708ed890_4992_4dab_9878_b000a9c3de79.slice/crio-30ab449b748455804f448471a88f37f285ef3131a5e995a8dc5eb303115e1779 WatchSource:0}: Error finding container 30ab449b748455804f448471a88f37f285ef3131a5e995a8dc5eb303115e1779: Status 404 returned error can't find the container with id 30ab449b748455804f448471a88f37f285ef3131a5e995a8dc5eb303115e1779
Nov 02 02:01:08.238053 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:08.238019    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e162987e-acf2-4901-8631-822c903e32d8" path="/var/lib/kubelet/pods/e162987e-acf2-4901-8631-822c903e32d8/volumes"
Nov 02 02:01:08.376247 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:08.376205    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs" event={"ID":"708ed890-4992-4dab-9878-b000a9c3de79","Type":"ContainerStarted","Data":"33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa"}
Nov 02 02:01:08.376446 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:08.376288    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:08.376446 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:08.376313    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs" event={"ID":"708ed890-4992-4dab-9878-b000a9c3de79","Type":"ContainerStarted","Data":"30ab449b748455804f448471a88f37f285ef3131a5e995a8dc5eb303115e1779"}
Nov 02 02:01:08.537262 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:08.537194    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs" podStartSLOduration=2.734642028 podStartE2EDuration="3.537176374s" totalImagesPullingTime="802.534346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:08.524069251 +0000 UTC m=+2176.397710303" watchObservedRunningTime="2026-11-02 02:01:08.537176374 +0000 UTC m=+2176.410817426"
Nov 02 02:01:14.389666 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:14.389621    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:01:16.080202 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:01:16.080148    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:01:21.570766 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:21.570724    4287 generic.go:356] "Generic (PLEG): container finished" podID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerID="f92f99adb09d6d904fb6866957ee8727dc5ce16a8de874f88287583d0b4d40f2" exitCode=0
Nov 02 02:01:21.570766 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:21.570748    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" event={"ID":"65c2e456-b896-4e5f-a696-7e26fabff35e","Type":"ContainerDied","Data":"f92f99adb09d6d904fb6866957ee8727dc5ce16a8de874f88287583d0b4d40f2"}
Nov 02 02:01:21.571230 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:21.571146    4287 scope.go:122] "RemoveContainer" containerID="f92f99adb09d6d904fb6866957ee8727dc5ce16a8de874f88287583d0b4d40f2"
Nov 02 02:01:21.572661 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:21.572635    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:21.572726 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:21.572704    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:21.572767 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:21.572733    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:01:22.582842 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:22.582804    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" event={"ID":"65c2e456-b896-4e5f-a696-7e26fabff35e","Type":"ContainerStarted","Data":"f55faa2402d777cae6f0ade6a7e06f0a3d3377a898cfa561730a805a339b1efa"}
Nov 02 02:01:23.410463 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:23.410426    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 02:01:23.410719 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:23.410674    4287 patch_prober.go:29] interesting pod/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 02:01:23.410870 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:23.410726    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 02:01:24.410655 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:24.410592    4287 patch_prober.go:29] interesting pod/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 02:01:24.411060 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:24.410685    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 02:01:25.410251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:25.410210    4287 patch_prober.go:29] interesting pod/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 02:01:25.410447 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:25.410279    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 02:01:26.410182 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:26.410149    4287 patch_prober.go:29] interesting pod/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 02:01:26.410583 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:26.410205    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 02:01:27.410491 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:27.410451    4287 patch_prober.go:29] interesting pod/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 02:01:27.410922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:27.410518    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 02:01:28.415234 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:28.415203    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 02:01:28.415743 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:28.415722    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 02:01:28.416531 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:28.416509    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 02:01:33.077054 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.077016    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"]
Nov 02 02:01:33.077524 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.077212    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq" podUID="5df408c8-a8fa-4c65-b307-1fc0c6c43c87" containerName="maas-api" containerID="cri-o://e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901" gracePeriod=30
Nov 02 02:01:33.393204 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.393177    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 02:01:33.547128 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.547091    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-metrics-certs\") pod \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") "
Nov 02 02:01:33.547355 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.547338    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-maas-api-tls\") pod \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") "
Nov 02 02:01:33.547482 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.547466    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-kube-api-access-jkq87\" (UniqueName: \"kubernetes.io/projected/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-kube-api-access-jkq87\") pod \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") "
Nov 02 02:01:33.547740 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.547718    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-openshift-service-ca\") pod \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\" (UID: \"5df408c8-a8fa-4c65-b307-1fc0c6c43c87\") "
Nov 02 02:01:33.549086 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.549052    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-openshift-service-ca" pod "5df408c8-a8fa-4c65-b307-1fc0c6c43c87" (UID: "5df408c8-a8fa-4c65-b307-1fc0c6c43c87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:01:33.603596 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.603512    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-maas-api-tls" pod "5df408c8-a8fa-4c65-b307-1fc0c6c43c87" (UID: "5df408c8-a8fa-4c65-b307-1fc0c6c43c87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:33.603963 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.603944    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-metrics-certs" pod "5df408c8-a8fa-4c65-b307-1fc0c6c43c87" (UID: "5df408c8-a8fa-4c65-b307-1fc0c6c43c87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:01:33.604288 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.604275    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-kube-api-access-jkq87" pod "5df408c8-a8fa-4c65-b307-1fc0c6c43c87" (UID: "5df408c8-a8fa-4c65-b307-1fc0c6c43c87"). InnerVolumeSpecName "kube-api-access-jkq87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:01:33.650085 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.650045    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:33.650085 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.650080    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:33.650300 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.650095    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkq87\" (UniqueName: \"kubernetes.io/projected/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-kube-api-access-jkq87\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:33.650300 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.650109    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df408c8-a8fa-4c65-b307-1fc0c6c43c87-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:01:33.655045 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.655015    4287 generic.go:356] "Generic (PLEG): container finished" podID="5df408c8-a8fa-4c65-b307-1fc0c6c43c87" containerID="e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901" exitCode=0
Nov 02 02:01:33.655195 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.655076    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"
Nov 02 02:01:33.655195 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.655106    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq" event={"ID":"5df408c8-a8fa-4c65-b307-1fc0c6c43c87","Type":"ContainerDied","Data":"e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901"}
Nov 02 02:01:33.655195 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.655165    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq" event={"ID":"5df408c8-a8fa-4c65-b307-1fc0c6c43c87","Type":"ContainerDied","Data":"2ba0ee8b2d5b71601e800bfb1ab14dff4ca84ec05922bb20a1a4e401f925224a"}
Nov 02 02:01:33.655336 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.655196    4287 scope.go:122] "RemoveContainer" containerID="e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901"
Nov 02 02:01:33.670069 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.670039    4287 scope.go:122] "RemoveContainer" containerID="e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901"
Nov 02 02:01:33.670530 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:01:33.670509    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901\": container with ID starting with e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901 not found: ID does not exist" containerID="e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901"
Nov 02 02:01:33.670634 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.670538    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901"} err="failed to get container status \"e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901\": rpc error: code = NotFound desc = could not find container \"e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901\": container with ID starting with e3beccb83c93371b3163ee62c843a39f12831e3e4cb7084d873cbae1cc282901 not found: ID does not exist"
Nov 02 02:01:33.821378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.821338    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"]
Nov 02 02:01:33.824816 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:33.824788    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c600fc-694c77fd46-vv8zq"]
Nov 02 02:01:34.239017 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:01:34.238985    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df408c8-a8fa-4c65-b307-1fc0c6c43c87" path="/var/lib/kubelet/pods/5df408c8-a8fa-4c65-b307-1fc0c6c43c87/volumes"
Nov 02 02:01:46.039105 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:01:46.039067    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:02:06.875666 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:06.875603    4287 generic.go:356] "Generic (PLEG): container finished" podID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerID="ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce" exitCode=0
Nov 02 02:02:06.876084 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:06.875636    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" event={"ID":"7415c203-cde7-40e1-a9a1-b6d3abaf7272","Type":"ContainerDied","Data":"ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce"}
Nov 02 02:02:06.876199 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:06.876181    4287 scope.go:122] "RemoveContainer" containerID="ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce"
Nov 02 02:02:06.880430 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:06.880398    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:06.880507 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:06.880468    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:06.880507 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:06.880501    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:02:07.885226 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:07.885193    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" event={"ID":"7415c203-cde7-40e1-a9a1-b6d3abaf7272","Type":"ContainerStarted","Data":"20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70"}
Nov 02 02:02:08.422394 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:08.422357    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 02:02:08.423695 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:08.423669    4287 patch_prober.go:29] interesting pod/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 02 02:02:08.423820 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:08.423715    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 02 02:02:08.475160 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:08.475125    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 02:02:09.422426 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:09.422393    4287 patch_prober.go:29] interesting pod/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 02 02:02:09.422857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:09.422446    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 02 02:02:10.422830 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:10.422798    4287 patch_prober.go:29] interesting pod/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 02 02:02:10.423206 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:10.422854    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 02 02:02:11.422401 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:11.422364    4287 patch_prober.go:29] interesting pod/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 02 02:02:11.422571 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:11.422419    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 02 02:02:12.422383 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:12.422350    4287 patch_prober.go:29] interesting pod/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 02 02:02:12.422776 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:12.422402    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 02 02:02:13.426174 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:13.426142    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 02:02:13.427343 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:13.427319    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 02:02:16.039176 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:02:16.039139    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:02:16.085424 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:16.085393    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"]
Nov 02 02:02:16.085680 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:16.085656    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" containerID="cri-o://cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6" gracePeriod=30
Nov 02 02:02:16.450416 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:16.450328    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"]
Nov 02 02:02:16.450602 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:16.450562    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy" containerID="cri-o://fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1" gracePeriod=30
Nov 02 02:02:16.453925 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:16.453901    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"]
Nov 02 02:02:16.454119 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:16.454095    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy" containerID="cri-o://cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84" gracePeriod=30
Nov 02 02:02:16.457122 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:16.457104    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"]
Nov 02 02:02:16.457279 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:16.457262    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy" containerID="cri-o://51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1" gracePeriod=30
Nov 02 02:02:18.035273 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:18.035234    4287 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:02:18.035792 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:18.035288    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:02:18.119874 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:18.117871    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"]
Nov 02 02:02:18.119874 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:18.118140    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy" containerID="cri-o://9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2" gracePeriod=30
Nov 02 02:02:18.722049 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:18.722012    4287 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:02:18.722222 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:18.722077    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:02:21.341158 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.341129    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 02:02:21.499734 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.499703    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istiod-ca-cert\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.499865 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.499780    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-socket\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.499865 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.499855    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-certs\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.499957 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.499902    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-credential-socket\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.499957 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.499938    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-token\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-token\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.500122 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.499968    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-podinfo\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.500122 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.499993    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-envoy\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.500122 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500027    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-data\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.500122 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500061    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-kube-api-access-r4cbb\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-kube-api-access-r4cbb\") pod \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\" (UID: \"dc1e58e2-df90-44dd-bf75-656c94f4bcdc\") "
Nov 02 02:02:21.500315 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500143    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-socket" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.500315 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500168    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istiod-ca-cert" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:21.500315 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500275    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-credential-socket" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.500691 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500485    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-data" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.500861 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500831    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-certs" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.500948 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500906    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.500948 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500930    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.500948 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500945    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.501282 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.500959    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.502619 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.502576    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-token" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:21.502916 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.502897    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-kube-api-access-r4cbb" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "kube-api-access-r4cbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:21.503006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.502992    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-envoy" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.503066 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.503019    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-podinfo" pod "dc1e58e2-df90-44dd-bf75-656c94f4bcdc" (UID: "dc1e58e2-df90-44dd-bf75-656c94f4bcdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:21.601902 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.601867    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.601902 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.601896    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.601902 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.601908    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.602211 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.601921    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.602211 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.601932    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4cbb\" (UniqueName: \"kubernetes.io/projected/dc1e58e2-df90-44dd-bf75-656c94f4bcdc-kube-api-access-r4cbb\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.721439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.721330    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 02:02:21.785746 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.785721    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 02:02:21.790665 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.790646    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 02:02:21.804384 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804360    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-data\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804512 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804416    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed9364f0-0590-4807-84ff-a579394bf398-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9364f0-0590-4807-84ff-a579394bf398-istio-podinfo\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804512 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804467    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-certs\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804512 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804499    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-kube-api-access-xsg82\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-kube-api-access-xsg82\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804699 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804570    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-envoy\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804699 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804641    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-credential-socket\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804699 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804682    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-istio-token\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-istio-token\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804699 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804690    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-data" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.804925 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804803    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-socket\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804925 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804860    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9364f0-0590-4807-84ff-a579394bf398-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9364f0-0590-4807-84ff-a579394bf398-istiod-ca-cert\") pod \"ed9364f0-0590-4807-84ff-a579394bf398\" (UID: \"ed9364f0-0590-4807-84ff-a579394bf398\") "
Nov 02 02:02:21.804925 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.804913    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-credential-socket" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.805480 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.805455    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.805568 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.805489    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.805568 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.805463    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-socket" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.805724 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.805698    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-certs" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.805945 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.805923    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9364f0-0590-4807-84ff-a579394bf398-istiod-ca-cert" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:21.807047 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.807015    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed9364f0-0590-4807-84ff-a579394bf398-istio-podinfo" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:21.807444 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.807414    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-kube-api-access-xsg82" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "kube-api-access-xsg82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:21.807444 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.807414    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-envoy" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.807978 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.807958    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-istio-token" pod "ed9364f0-0590-4807-84ff-a579394bf398" (UID: "ed9364f0-0590-4807-84ff-a579394bf398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:21.906016 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.905987    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istiod-ca-cert\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.906185 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906035    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7862ab77-e085-41b1-b594-44a22aa6b640-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ab77-e085-41b1-b594-44a22aa6b640-istio-podinfo\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.906185 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906067    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-socket\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.906185 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906122    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-certs\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.906358 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906205    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-credential-socket\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.906358 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906233    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-kube-api-access-9rrhg\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-kube-api-access-9rrhg\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.906358 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906274    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-data\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.906358 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906305    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-envoy\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.906576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906359    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-istio-token\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-istio-token\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.906576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906363    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-socket" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.906576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906396    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-credential-socket\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.906576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906423    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-podinfo\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.906576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906454    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-envoy\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.906576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906495    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-certs" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.906576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906402    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istiod-ca-cert" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:21.906576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906550    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-data\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.907009 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906583    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7862ab77-e085-41b1-b594-44a22aa6b640-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ab77-e085-41b1-b594-44a22aa6b640-istiod-ca-cert\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.907009 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906641    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-certs\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.907009 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906672    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-kube-api-access-fhtbd\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-kube-api-access-fhtbd\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.907009 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906677    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-data" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.907009 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906732    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-token\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-token\") pod \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\" (UID: \"d8276d1e-3e79-4fa2-8688-14f3da5c3a92\") "
Nov 02 02:02:21.907009 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906760    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-socket\") pod \"7862ab77-e085-41b1-b594-44a22aa6b640\" (UID: \"7862ab77-e085-41b1-b594-44a22aa6b640\") "
Nov 02 02:02:21.907009 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906746    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-credential-socket" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.907009 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.906790    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-credential-socket" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.907421 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907125    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7862ab77-e085-41b1-b594-44a22aa6b640-istiod-ca-cert" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:21.907421 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907338    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907421 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907355    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-data" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.907421 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907359    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907421 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907397    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9364f0-0590-4807-84ff-a579394bf398-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907421 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907415    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907430    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsg82\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-kube-api-access-xsg82\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907446    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907460    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907476    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9364f0-0590-4807-84ff-a579394bf398-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907490    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907533    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907548    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9364f0-0590-4807-84ff-a579394bf398-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907555    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-certs" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907562    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7862ab77-e085-41b1-b594-44a22aa6b640-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907593    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9364f0-0590-4807-84ff-a579394bf398-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.907907 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.907650    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:21.908671 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.908010    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-socket" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.909165 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.909132    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7862ab77-e085-41b1-b594-44a22aa6b640-istio-podinfo" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:21.909284 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.909194    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-istio-token" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:21.909830 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.909809    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-envoy" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.909921 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.909848    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-token" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:21.909961 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.909928    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-podinfo" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:21.909996 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.909956    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-kube-api-access-9rrhg" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "kube-api-access-9rrhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:21.910273 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.910249    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-kube-api-access-fhtbd" pod "d8276d1e-3e79-4fa2-8688-14f3da5c3a92" (UID: "d8276d1e-3e79-4fa2-8688-14f3da5c3a92"). InnerVolumeSpecName "kube-api-access-fhtbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:21.910570 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.910552    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-envoy" pod "7862ab77-e085-41b1-b594-44a22aa6b640" (UID: "7862ab77-e085-41b1-b594-44a22aa6b640"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:21.991674 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.991553    4287 generic.go:356] "Generic (PLEG): container finished" podID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerID="cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6" exitCode=0
Nov 02 02:02:21.991821 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.991747    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" event={"ID":"dc1e58e2-df90-44dd-bf75-656c94f4bcdc","Type":"ContainerDied","Data":"cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6"}
Nov 02 02:02:21.991821 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.991809    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56" event={"ID":"dc1e58e2-df90-44dd-bf75-656c94f4bcdc","Type":"ContainerDied","Data":"a03b8d43d35afa1e11ad52991b3c674419ae9f4e53189bbc36f9e8946a04af89"}
Nov 02 02:02:21.991905 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.991841    4287 scope.go:122] "RemoveContainer" containerID="cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6"
Nov 02 02:02:21.992011 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:21.991995    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"
Nov 02 02:02:22.002799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.000220    4287 generic.go:356] "Generic (PLEG): container finished" podID="7862ab77-e085-41b1-b594-44a22aa6b640" containerID="51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1" exitCode=0
Nov 02 02:02:22.002799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.000282    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"
Nov 02 02:02:22.002799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.000300    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" event={"ID":"7862ab77-e085-41b1-b594-44a22aa6b640","Type":"ContainerDied","Data":"51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1"}
Nov 02 02:02:22.002799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.000355    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk" event={"ID":"7862ab77-e085-41b1-b594-44a22aa6b640","Type":"ContainerDied","Data":"c869aaf8d050f768735c791f519236fc4f73e74ae53032c2dc8b2ce94983ef6d"}
Nov 02 02:02:22.002799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.002031    4287 generic.go:356] "Generic (PLEG): container finished" podID="ed9364f0-0590-4807-84ff-a579394bf398" containerID="fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1" exitCode=0
Nov 02 02:02:22.002799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.002075    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"
Nov 02 02:02:22.002799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.002162    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" event={"ID":"ed9364f0-0590-4807-84ff-a579394bf398","Type":"ContainerDied","Data":"fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1"}
Nov 02 02:02:22.002799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.002212    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h" event={"ID":"ed9364f0-0590-4807-84ff-a579394bf398","Type":"ContainerDied","Data":"432ba7d907ce148f44b7793b15472bb228a16734620bac091a6f963d335253f9"}
Nov 02 02:02:22.005645 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.005594    4287 generic.go:356] "Generic (PLEG): container finished" podID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerID="cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84" exitCode=0
Nov 02 02:02:22.005745 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.005652    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" event={"ID":"d8276d1e-3e79-4fa2-8688-14f3da5c3a92","Type":"ContainerDied","Data":"cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84"}
Nov 02 02:02:22.005745 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.005678    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"
Nov 02 02:02:22.005745 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.005702    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42" event={"ID":"d8276d1e-3e79-4fa2-8688-14f3da5c3a92","Type":"ContainerDied","Data":"7497cfedf2565eb8f5d00ea6d50e46ee734285837c7b6686f00ebea83ac1455c"}
Nov 02 02:02:22.007877 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.007857    4287 scope.go:122] "RemoveContainer" containerID="cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6"
Nov 02 02:02:22.008168 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:02:22.008140    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6\": container with ID starting with cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6 not found: ID does not exist" containerID="cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6"
Nov 02 02:02:22.008243 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008180    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6"} err="failed to get container status \"cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6\": rpc error: code = NotFound desc = could not find container \"cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6\": container with ID starting with cde867ad2724326ffc15a6eede8b7003d35d603fe46331e6e6f5c6b7621c7bb6 not found: ID does not exist"
Nov 02 02:02:22.008243 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008202    4287 scope.go:122] "RemoveContainer" containerID="51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1"
Nov 02 02:02:22.008638 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008593    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrhg\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-kube-api-access-9rrhg\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.008749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008641    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.008749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008659    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7862ab77-e085-41b1-b594-44a22aa6b640-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.008749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008673    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.008749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008687    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.008749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008702    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.008749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008715    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.008749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008729    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhtbd\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-kube-api-access-fhtbd\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.008749 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008742    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8276d1e-3e79-4fa2-8688-14f3da5c3a92-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.009154 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008756    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7862ab77-e085-41b1-b594-44a22aa6b640-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.009154 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.008770    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7862ab77-e085-41b1-b594-44a22aa6b640-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:22.019496 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.019474    4287 scope.go:122] "RemoveContainer" containerID="51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1"
Nov 02 02:02:22.019846 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:02:22.019822    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1\": container with ID starting with 51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1 not found: ID does not exist" containerID="51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1"
Nov 02 02:02:22.019942 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.019856    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1"} err="failed to get container status \"51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1\": rpc error: code = NotFound desc = could not find container \"51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1\": container with ID starting with 51f0a90d4105b717cc403e75ebea048e598fd235387cc794ccc81d653e8b18c1 not found: ID does not exist"
Nov 02 02:02:22.019942 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.019879    4287 scope.go:122] "RemoveContainer" containerID="fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1"
Nov 02 02:02:22.029471 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.029455    4287 scope.go:122] "RemoveContainer" containerID="fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1"
Nov 02 02:02:22.029761 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:02:22.029743    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1\": container with ID starting with fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1 not found: ID does not exist" containerID="fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1"
Nov 02 02:02:22.029834 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.029767    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1"} err="failed to get container status \"fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1\": rpc error: code = NotFound desc = could not find container \"fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1\": container with ID starting with fd58e6f4b9f46bcc1d4ad18a9b41afb19e208ab1da59237b218d3c38a76a70b1 not found: ID does not exist"
Nov 02 02:02:22.029834 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.029786    4287 scope.go:122] "RemoveContainer" containerID="cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84"
Nov 02 02:02:22.038696 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.038676    4287 scope.go:122] "RemoveContainer" containerID="cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84"
Nov 02 02:02:22.038962 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:02:22.038942    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84\": container with ID starting with cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84 not found: ID does not exist" containerID="cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84"
Nov 02 02:02:22.039027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.038973    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84"} err="failed to get container status \"cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84\": rpc error: code = NotFound desc = could not find container \"cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84\": container with ID starting with cc4d1cb0d16bb317ad382648b2369e4d0fe765af47068e1bff6c839eb301df84 not found: ID does not exist"
Nov 02 02:02:22.202714 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.202669    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"]
Nov 02 02:02:22.205860 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.205840    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-g77jk"]
Nov 02 02:02:22.237860 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.237827    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" path="/var/lib/kubelet/pods/7862ab77-e085-41b1-b594-44a22aa6b640/volumes"
Nov 02 02:02:22.315008 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.314975    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"]
Nov 02 02:02:22.319934 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.319907    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-l7n56"]
Nov 02 02:02:22.430500 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.430461    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"]
Nov 02 02:02:22.433439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.433413    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-85v42"]
Nov 02 02:02:22.569155 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.569076    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"]
Nov 02 02:02:22.653587 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:22.649905    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-pls2h"]
Nov 02 02:02:23.384184 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.384157    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 02:02:23.526427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526391    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istiod-ca-cert\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526460    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-envoy\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526514    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-credential-socket\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526545    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-data\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526737    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-kube-api-access-bqxq5\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-kube-api-access-bqxq5\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526775    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-token\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-token\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526815    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-certs\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526873    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-socket\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526881    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-data" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526940    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istiod-ca-cert" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:23.526991 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.526989    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-podinfo\") pod \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\" (UID: \"2a4390e3-7a12-48cf-ae27-01e87d95d0d6\") "
Nov 02 02:02:23.527542 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.527113    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-socket" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:23.527542 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.527320    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-certs" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:23.527754 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.527730    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-credential-socket" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:23.527827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.527748    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:23.527827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.527772    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:23.527827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.527788    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:23.527827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.527802    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:23.529236 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.529206    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-kube-api-access-bqxq5" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "kube-api-access-bqxq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:23.529556 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.529527    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-envoy" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:02:23.529712 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.529634    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-token" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:23.529929 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.529901    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-podinfo" pod "2a4390e3-7a12-48cf-ae27-01e87d95d0d6" (UID: "2a4390e3-7a12-48cf-ae27-01e87d95d0d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:02:23.628748 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.628714    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:23.628748 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.628741    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:23.628748 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.628750    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:23.628973 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.628758    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:23.628973 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:23.628768    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqxq5\" (UniqueName: \"kubernetes.io/projected/2a4390e3-7a12-48cf-ae27-01e87d95d0d6-kube-api-access-bqxq5\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:24.043245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.034523    4287 generic.go:356] "Generic (PLEG): container finished" podID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerID="9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2" exitCode=0
Nov 02 02:02:24.043245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.034793    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" event={"ID":"2a4390e3-7a12-48cf-ae27-01e87d95d0d6","Type":"ContainerDied","Data":"9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2"}
Nov 02 02:02:24.043245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.034850    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr" event={"ID":"2a4390e3-7a12-48cf-ae27-01e87d95d0d6","Type":"ContainerDied","Data":"5a5057a52674524de803ce0b6595ebe968afa3a09f40e6b37e3c54d4c72a99ce"}
Nov 02 02:02:24.043245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.034876    4287 scope.go:122] "RemoveContainer" containerID="9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2"
Nov 02 02:02:24.043245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.035045    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"
Nov 02 02:02:24.060683 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.060589    4287 scope.go:122] "RemoveContainer" containerID="9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2"
Nov 02 02:02:24.061088 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:02:24.061045    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2\": container with ID starting with 9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2 not found: ID does not exist" containerID="9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2"
Nov 02 02:02:24.061189 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.061087    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2"} err="failed to get container status \"9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2\": rpc error: code = NotFound desc = could not find container \"9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2\": container with ID starting with 9293cb1e2633af33da0f63f5f3715b33c0b85c1eddce6701cb47d6fb256f39b2 not found: ID does not exist"
Nov 02 02:02:24.237684 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.237653    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" path="/var/lib/kubelet/pods/d8276d1e-3e79-4fa2-8688-14f3da5c3a92/volumes"
Nov 02 02:02:24.238165 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.238149    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" path="/var/lib/kubelet/pods/dc1e58e2-df90-44dd-bf75-656c94f4bcdc/volumes"
Nov 02 02:02:24.238642 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.238626    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9364f0-0590-4807-84ff-a579394bf398" path="/var/lib/kubelet/pods/ed9364f0-0590-4807-84ff-a579394bf398/volumes"
Nov 02 02:02:24.253669 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.253638    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"]
Nov 02 02:02:24.302705 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:24.302636    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xspvr"]
Nov 02 02:02:26.238163 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:26.238131    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" path="/var/lib/kubelet/pods/2a4390e3-7a12-48cf-ae27-01e87d95d0d6/volumes"
Nov 02 02:02:46.039418 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:02:46.039377    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:02:47.702166 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:47.702124    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"]
Nov 02 02:02:47.702595 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:47.702379    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4" podUID="21073d91-773c-4ff8-9486-4b78a3c177a3" containerName="maas-api" containerID="cri-o://ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8" gracePeriod=30
Nov 02 02:02:47.990751 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:47.990725    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 02:02:48.144745 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.144715    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21073d91-773c-4ff8-9486-4b78a3c177a3-kube-api-access-8wxj2\" (UniqueName: \"kubernetes.io/projected/21073d91-773c-4ff8-9486-4b78a3c177a3-kube-api-access-8wxj2\") pod \"21073d91-773c-4ff8-9486-4b78a3c177a3\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") "
Nov 02 02:02:48.144924 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.144875    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-metrics-certs\") pod \"21073d91-773c-4ff8-9486-4b78a3c177a3\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") "
Nov 02 02:02:48.144995 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.144939    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21073d91-773c-4ff8-9486-4b78a3c177a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21073d91-773c-4ff8-9486-4b78a3c177a3-openshift-service-ca\") pod \"21073d91-773c-4ff8-9486-4b78a3c177a3\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") "
Nov 02 02:02:48.145102 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.145048    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls\") pod \"21073d91-773c-4ff8-9486-4b78a3c177a3\" (UID: \"21073d91-773c-4ff8-9486-4b78a3c177a3\") "
Nov 02 02:02:48.145491 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.145459    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21073d91-773c-4ff8-9486-4b78a3c177a3-openshift-service-ca" pod "21073d91-773c-4ff8-9486-4b78a3c177a3" (UID: "21073d91-773c-4ff8-9486-4b78a3c177a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:02:48.145746 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.145687    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21073d91-773c-4ff8-9486-4b78a3c177a3-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:48.147406 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.147356    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21073d91-773c-4ff8-9486-4b78a3c177a3-kube-api-access-8wxj2" pod "21073d91-773c-4ff8-9486-4b78a3c177a3" (UID: "21073d91-773c-4ff8-9486-4b78a3c177a3"). InnerVolumeSpecName "kube-api-access-8wxj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:02:48.147978 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.147951    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-metrics-certs" pod "21073d91-773c-4ff8-9486-4b78a3c177a3" (UID: "21073d91-773c-4ff8-9486-4b78a3c177a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:48.148236 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.148207    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls" pod "21073d91-773c-4ff8-9486-4b78a3c177a3" (UID: "21073d91-773c-4ff8-9486-4b78a3c177a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:02:48.218391 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.218305    4287 generic.go:356] "Generic (PLEG): container finished" podID="21073d91-773c-4ff8-9486-4b78a3c177a3" containerID="ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8" exitCode=0
Nov 02 02:02:48.218391 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.218359    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"
Nov 02 02:02:48.218563 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.218395    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4" event={"ID":"21073d91-773c-4ff8-9486-4b78a3c177a3","Type":"ContainerDied","Data":"ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8"}
Nov 02 02:02:48.218563 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.218457    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4" event={"ID":"21073d91-773c-4ff8-9486-4b78a3c177a3","Type":"ContainerDied","Data":"ad22b30816399afe78cad53372395b7080bfb41a9f386e8a4e74dc9a82220c56"}
Nov 02 02:02:48.218563 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.218481    4287 scope.go:122] "RemoveContainer" containerID="ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8"
Nov 02 02:02:48.229334 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.229317    4287 scope.go:122] "RemoveContainer" containerID="ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8"
Nov 02 02:02:48.229601 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:02:48.229579    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8\": container with ID starting with ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8 not found: ID does not exist" containerID="ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8"
Nov 02 02:02:48.229728 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.229625    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8"} err="failed to get container status \"ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8\": rpc error: code = NotFound desc = could not find container \"ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8\": container with ID starting with ca0ae987a1006048830805450785654fc406971878fcdb1fdb292865821568e8 not found: ID does not exist"
Nov 02 02:02:48.246940 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.246919    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:48.247027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.246944    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wxj2\" (UniqueName: \"kubernetes.io/projected/21073d91-773c-4ff8-9486-4b78a3c177a3-kube-api-access-8wxj2\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:48.247027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.246958    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21073d91-773c-4ff8-9486-4b78a3c177a3-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:02:48.390834 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.390799    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"]
Nov 02 02:02:48.394366 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:48.394343    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-19436b-844d577b88-x7dx4"]
Nov 02 02:02:50.237094 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:50.237058    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21073d91-773c-4ff8-9486-4b78a3c177a3" path="/var/lib/kubelet/pods/21073d91-773c-4ff8-9486-4b78a3c177a3/volumes"
Nov 02 02:02:59.391461 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:02:59.391424    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"]
Nov 02 02:03:14.416202 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.416155    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main" containerID="cri-o://7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d" gracePeriod=45
Nov 02 02:03:14.687812 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.687783    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 02:03:14.800487 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800453    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-model-cache\") pod \"950028ae-303e-476d-9271-052f2bd8db10\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") "
Nov 02 02:03:14.800703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800504    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-dshm\") pod \"950028ae-303e-476d-9271-052f2bd8db10\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") "
Nov 02 02:03:14.800703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800570    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-tmp-dir\") pod \"950028ae-303e-476d-9271-052f2bd8db10\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") "
Nov 02 02:03:14.800703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800651    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-home\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-home\") pod \"950028ae-303e-476d-9271-052f2bd8db10\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") "
Nov 02 02:03:14.800703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800696    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/950028ae-303e-476d-9271-052f2bd8db10-tls-certs\" (UniqueName: \"kubernetes.io/secret/950028ae-303e-476d-9271-052f2bd8db10-tls-certs\") pod \"950028ae-303e-476d-9271-052f2bd8db10\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") "
Nov 02 02:03:14.800936 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800744    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950028ae-303e-476d-9271-052f2bd8db10-kube-api-access-25krh\" (UniqueName: \"kubernetes.io/projected/950028ae-303e-476d-9271-052f2bd8db10-kube-api-access-25krh\") pod \"950028ae-303e-476d-9271-052f2bd8db10\" (UID: \"950028ae-303e-476d-9271-052f2bd8db10\") "
Nov 02 02:03:14.800936 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800752    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-model-cache" pod "950028ae-303e-476d-9271-052f2bd8db10" (UID: "950028ae-303e-476d-9271-052f2bd8db10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:14.800936 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800915    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-home" pod "950028ae-303e-476d-9271-052f2bd8db10" (UID: "950028ae-303e-476d-9271-052f2bd8db10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:14.801080 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.800933    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-tmp-dir" pod "950028ae-303e-476d-9271-052f2bd8db10" (UID: "950028ae-303e-476d-9271-052f2bd8db10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:14.801291 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.801267    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.801580 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.801293    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.801580 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.801308    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.803010 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.802982    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-dshm" pod "950028ae-303e-476d-9271-052f2bd8db10" (UID: "950028ae-303e-476d-9271-052f2bd8db10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:14.803092 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.803013    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950028ae-303e-476d-9271-052f2bd8db10-kube-api-access-25krh" pod "950028ae-303e-476d-9271-052f2bd8db10" (UID: "950028ae-303e-476d-9271-052f2bd8db10"). InnerVolumeSpecName "kube-api-access-25krh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:14.803092 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.803051    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950028ae-303e-476d-9271-052f2bd8db10-tls-certs" pod "950028ae-303e-476d-9271-052f2bd8db10" (UID: "950028ae-303e-476d-9271-052f2bd8db10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:14.902289 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.902256    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950028ae-303e-476d-9271-052f2bd8db10-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.902289 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.902286    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950028ae-303e-476d-9271-052f2bd8db10-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:14.902289 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:14.902296    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25krh\" (UniqueName: \"kubernetes.io/projected/950028ae-303e-476d-9271-052f2bd8db10-kube-api-access-25krh\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:15.397323 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.397284    4287 generic.go:356] "Generic (PLEG): container finished" podID="950028ae-303e-476d-9271-052f2bd8db10" containerID="7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d" exitCode=0
Nov 02 02:03:15.397323 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.397332    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"
Nov 02 02:03:15.397548 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.397340    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" event={"ID":"950028ae-303e-476d-9271-052f2bd8db10","Type":"ContainerDied","Data":"7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d"}
Nov 02 02:03:15.397548 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.397379    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp" event={"ID":"950028ae-303e-476d-9271-052f2bd8db10","Type":"ContainerDied","Data":"5a484736a05787f3d15288e34e53dadedb6c7358edef534a139592625ac96b09"}
Nov 02 02:03:15.397548 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.397400    4287 scope.go:122] "RemoveContainer" containerID="7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d"
Nov 02 02:03:15.410474 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.410436    4287 scope.go:122] "RemoveContainer" containerID="7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d"
Nov 02 02:03:15.411089 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:15.411065    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d\": container with ID starting with 7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d not found: ID does not exist" containerID="7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d"
Nov 02 02:03:15.411153 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.411103    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d"} err="failed to get container status \"7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d\": rpc error: code = NotFound desc = could not find container \"7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d\": container with ID starting with 7550931d30c2771b13206228eb714314a732b7319e7ca681665336ca7761317d not found: ID does not exist"
Nov 02 02:03:15.563106 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.563065    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"]
Nov 02 02:03:15.613537 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:15.613498    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-2-w4-187478-kserve-6ddd9bf9ff-bj5qp"]
Nov 02 02:03:16.039044 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:16.039002    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:03:16.243716 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:16.243684    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950028ae-303e-476d-9271-052f2bd8db10" path="/var/lib/kubelet/pods/950028ae-303e-476d-9271-052f2bd8db10/volumes"
Nov 02 02:03:25.215789 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.215754    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"]
Nov 02 02:03:25.216162 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.215977    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz" podUID="4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" containerName="maas-api" containerID="cri-o://a235768bb8224810b8fd7bac90fdd3dd501fead529bd06996f038f249440baee" gracePeriod=30
Nov 02 02:03:25.490390 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.490361    4287 generic.go:356] "Generic (PLEG): container finished" podID="4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" containerID="a235768bb8224810b8fd7bac90fdd3dd501fead529bd06996f038f249440baee" exitCode=0
Nov 02 02:03:25.490515 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.490502    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz" event={"ID":"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1","Type":"ContainerDied","Data":"a235768bb8224810b8fd7bac90fdd3dd501fead529bd06996f038f249440baee"}
Nov 02 02:03:25.490567 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.490541    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz" event={"ID":"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1","Type":"ContainerDied","Data":"f48624b9258e992b6c395b26ce2b0bb8306a422bf75f0902920c20974c88b3d6"}
Nov 02 02:03:25.490567 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.490557    4287 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48624b9258e992b6c395b26ce2b0bb8306a422bf75f0902920c20974c88b3d6"
Nov 02 02:03:25.512188 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.512168    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:03:25.639097 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.639056    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-openshift-service-ca\") pod \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") "
Nov 02 02:03:25.639381 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.639351    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls\") pod \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") "
Nov 02 02:03:25.640070 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.640048    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-kube-api-access-vnw2j\" (UniqueName: \"kubernetes.io/projected/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-kube-api-access-vnw2j\") pod \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") "
Nov 02 02:03:25.640330 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.640312    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-metrics-certs\") pod \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\" (UID: \"4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1\") "
Nov 02 02:03:25.646348 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.646297    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-openshift-service-ca" pod "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" (UID: "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:25.647841 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.647788    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls" pod "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" (UID: "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:25.651116 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.651078    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-kube-api-access-vnw2j" pod "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" (UID: "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1"). InnerVolumeSpecName "kube-api-access-vnw2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:25.651432 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.651399    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-metrics-certs" pod "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" (UID: "4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:25.743272 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.743172    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnw2j\" (UniqueName: \"kubernetes.io/projected/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-kube-api-access-vnw2j\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:25.743272 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.743214    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:25.743272 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.743228    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:25.743272 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:25.743240    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:26.496889 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:26.496852    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"
Nov 02 02:03:26.661131 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:26.661090    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"]
Nov 02 02:03:26.710487 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:26.710454    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-269235-787486f758-hqkqz"]
Nov 02 02:03:28.237741 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:28.237699    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" path="/var/lib/kubelet/pods/4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1/volumes"
Nov 02 02:03:31.414405 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.414370    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"]
Nov 02 02:03:31.414872 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.414577    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs" podUID="708ed890-4992-4dab-9878-b000a9c3de79" containerName="maas-api" containerID="cri-o://33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa" gracePeriod=30
Nov 02 02:03:31.726680 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.726654    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:03:31.805627 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.804525    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls\") pod \"708ed890-4992-4dab-9878-b000a9c3de79\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") "
Nov 02 02:03:31.805627 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.804668    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708ed890-4992-4dab-9878-b000a9c3de79-kube-api-access-vqqf5\" (UniqueName: \"kubernetes.io/projected/708ed890-4992-4dab-9878-b000a9c3de79-kube-api-access-vqqf5\") pod \"708ed890-4992-4dab-9878-b000a9c3de79\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") "
Nov 02 02:03:31.805627 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.804722    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708ed890-4992-4dab-9878-b000a9c3de79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708ed890-4992-4dab-9878-b000a9c3de79-openshift-service-ca\") pod \"708ed890-4992-4dab-9878-b000a9c3de79\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") "
Nov 02 02:03:31.805627 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.804823    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-metrics-certs\") pod \"708ed890-4992-4dab-9878-b000a9c3de79\" (UID: \"708ed890-4992-4dab-9878-b000a9c3de79\") "
Nov 02 02:03:31.814862 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.814809    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708ed890-4992-4dab-9878-b000a9c3de79-openshift-service-ca" pod "708ed890-4992-4dab-9878-b000a9c3de79" (UID: "708ed890-4992-4dab-9878-b000a9c3de79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:31.826894 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.826841    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-metrics-certs" pod "708ed890-4992-4dab-9878-b000a9c3de79" (UID: "708ed890-4992-4dab-9878-b000a9c3de79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:31.835691 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.835637    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls" pod "708ed890-4992-4dab-9878-b000a9c3de79" (UID: "708ed890-4992-4dab-9878-b000a9c3de79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:31.854829 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.854402    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708ed890-4992-4dab-9878-b000a9c3de79-kube-api-access-vqqf5" pod "708ed890-4992-4dab-9878-b000a9c3de79" (UID: "708ed890-4992-4dab-9878-b000a9c3de79"). InnerVolumeSpecName "kube-api-access-vqqf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:31.906403 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.906362    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqf5\" (UniqueName: \"kubernetes.io/projected/708ed890-4992-4dab-9878-b000a9c3de79-kube-api-access-vqqf5\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:31.906403 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.906391    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708ed890-4992-4dab-9878-b000a9c3de79-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:31.906403 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.906403    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:31.906403 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:31.906411    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708ed890-4992-4dab-9878-b000a9c3de79-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:32.535286 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.535254    4287 generic.go:356] "Generic (PLEG): container finished" podID="708ed890-4992-4dab-9878-b000a9c3de79" containerID="33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa" exitCode=0
Nov 02 02:03:32.535746 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.535371    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"
Nov 02 02:03:32.535746 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.535397    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs" event={"ID":"708ed890-4992-4dab-9878-b000a9c3de79","Type":"ContainerDied","Data":"33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa"}
Nov 02 02:03:32.535746 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.535441    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs" event={"ID":"708ed890-4992-4dab-9878-b000a9c3de79","Type":"ContainerDied","Data":"30ab449b748455804f448471a88f37f285ef3131a5e995a8dc5eb303115e1779"}
Nov 02 02:03:32.535746 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.535461    4287 scope.go:122] "RemoveContainer" containerID="33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa"
Nov 02 02:03:32.545186 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.545167    4287 scope.go:122] "RemoveContainer" containerID="33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa"
Nov 02 02:03:32.545472 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:32.545453    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa\": container with ID starting with 33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa not found: ID does not exist" containerID="33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa"
Nov 02 02:03:32.545522 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.545487    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa"} err="failed to get container status \"33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa\": rpc error: code = NotFound desc = could not find container \"33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa\": container with ID starting with 33a195f05c8ce718fd2efbea9d430fe7bf65bc18e946853b32b8bd73b338e1aa not found: ID does not exist"
Nov 02 02:03:32.581063 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.581032    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"]
Nov 02 02:03:32.581237 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.581218    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn" podUID="712e00b8-0705-43e4-ad47-a2e79e00e4fd" containerName="maas-api" containerID="cri-o://ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4" gracePeriod=30
Nov 02 02:03:32.744026 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.743994    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"]
Nov 02 02:03:32.751786 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.751754    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-69c15d-5488dd9dd-vwhgs"]
Nov 02 02:03:32.910597 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.910563    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 02:03:32.927828 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.927793    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-metrics-certs\") pod \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") "
Nov 02 02:03:32.927981 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.927853    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls\") pod \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") "
Nov 02 02:03:32.928041 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.927980    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712e00b8-0705-43e4-ad47-a2e79e00e4fd-kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/712e00b8-0705-43e4-ad47-a2e79e00e4fd-kube-api-access-5hnf8\") pod \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") "
Nov 02 02:03:32.928041 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.928016    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712e00b8-0705-43e4-ad47-a2e79e00e4fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e00b8-0705-43e4-ad47-a2e79e00e4fd-openshift-service-ca\") pod \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\" (UID: \"712e00b8-0705-43e4-ad47-a2e79e00e4fd\") "
Nov 02 02:03:32.928421 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.928387    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712e00b8-0705-43e4-ad47-a2e79e00e4fd-openshift-service-ca" pod "712e00b8-0705-43e4-ad47-a2e79e00e4fd" (UID: "712e00b8-0705-43e4-ad47-a2e79e00e4fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:32.928541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.928488    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712e00b8-0705-43e4-ad47-a2e79e00e4fd-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:32.930129 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.930108    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-metrics-certs" pod "712e00b8-0705-43e4-ad47-a2e79e00e4fd" (UID: "712e00b8-0705-43e4-ad47-a2e79e00e4fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:32.930317 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.930293    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712e00b8-0705-43e4-ad47-a2e79e00e4fd-kube-api-access-5hnf8" pod "712e00b8-0705-43e4-ad47-a2e79e00e4fd" (UID: "712e00b8-0705-43e4-ad47-a2e79e00e4fd"). InnerVolumeSpecName "kube-api-access-5hnf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:32.930708 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:32.930683    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls" pod "712e00b8-0705-43e4-ad47-a2e79e00e4fd" (UID: "712e00b8-0705-43e4-ad47-a2e79e00e4fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:03:33.036630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.036526    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:33.036630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.036569    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/712e00b8-0705-43e4-ad47-a2e79e00e4fd-kube-api-access-5hnf8\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:33.036630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.036585    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712e00b8-0705-43e4-ad47-a2e79e00e4fd-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:33.557805 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.557758    4287 generic.go:356] "Generic (PLEG): container finished" podID="712e00b8-0705-43e4-ad47-a2e79e00e4fd" containerID="ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4" exitCode=0
Nov 02 02:03:33.565636 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.558542    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn" event={"ID":"712e00b8-0705-43e4-ad47-a2e79e00e4fd","Type":"ContainerDied","Data":"ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4"}
Nov 02 02:03:33.566383 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.566353    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn" event={"ID":"712e00b8-0705-43e4-ad47-a2e79e00e4fd","Type":"ContainerDied","Data":"38be1516736176874f866e2246371a3955187648fa19795820ab1d434a8ac64d"}
Nov 02 02:03:33.566599 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.563768    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"
Nov 02 02:03:33.593632 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.566542    4287 scope.go:122] "RemoveContainer" containerID="ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4"
Nov 02 02:03:33.614538 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.614511    4287 scope.go:122] "RemoveContainer" containerID="ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4"
Nov 02 02:03:33.615102 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:33.615073    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4\": container with ID starting with ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4 not found: ID does not exist" containerID="ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4"
Nov 02 02:03:33.615195 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.615102    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4"} err="failed to get container status \"ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4\": rpc error: code = NotFound desc = could not find container \"ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4\": container with ID starting with ebce6ecd84e78a43f48b0b43b5f6eb95d6eeebf6960fba22a63b38ba996d2cb4 not found: ID does not exist"
Nov 02 02:03:33.743448 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.743410    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"]
Nov 02 02:03:33.747295 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:33.747271    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ef5a72-57d889c9c8-lbcgn"]
Nov 02 02:03:34.237919 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:34.237884    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708ed890-4992-4dab-9878-b000a9c3de79" path="/var/lib/kubelet/pods/708ed890-4992-4dab-9878-b000a9c3de79/volumes"
Nov 02 02:03:34.238400 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:34.238379    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712e00b8-0705-43e4-ad47-a2e79e00e4fd" path="/var/lib/kubelet/pods/712e00b8-0705-43e4-ad47-a2e79e00e4fd/volumes"
Nov 02 02:03:34.482483 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:34.482451    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"]
Nov 02 02:03:34.482718 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:34.482691    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy" containerID="cri-o://f55faa2402d777cae6f0ade6a7e06f0a3d3377a898cfa561730a805a339b1efa" gracePeriod=30
Nov 02 02:03:39.676691 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.676654    4287 generic.go:356] "Generic (PLEG): container finished" podID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerID="f55faa2402d777cae6f0ade6a7e06f0a3d3377a898cfa561730a805a339b1efa" exitCode=0
Nov 02 02:03:39.677089 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.676723    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" event={"ID":"65c2e456-b896-4e5f-a696-7e26fabff35e","Type":"ContainerDied","Data":"f55faa2402d777cae6f0ade6a7e06f0a3d3377a898cfa561730a805a339b1efa"}
Nov 02 02:03:39.677089 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.676806    4287 scope.go:122] "RemoveContainer" containerID="f92f99adb09d6d904fb6866957ee8727dc5ce16a8de874f88287583d0b4d40f2"
Nov 02 02:03:39.759534 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.759480    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 02:03:39.818923 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.818894    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-kube-api-access-vz6fg\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-kube-api-access-vz6fg\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819108 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819052    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-envoy\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819174 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819126    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-certs\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819225 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819203    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-podinfo\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819275 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819240    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-data\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819378 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819358    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c2e456-b896-4e5f-a696-7e26fabff35e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c2e456-b896-4e5f-a696-7e26fabff35e-istiod-ca-cert\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819440 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819403    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-credential-socket\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819512 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819447    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-token\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-token\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819660 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819627    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-socket\") pod \"65c2e456-b896-4e5f-a696-7e26fabff35e\" (UID: \"65c2e456-b896-4e5f-a696-7e26fabff35e\") "
Nov 02 02:03:39.819721 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819685    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-certs" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:39.820113 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.819990    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-socket" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:39.820390 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.820174    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-credential-socket" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:39.820684 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.820509    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-data" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:39.820684 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.820548    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:39.820684 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.820574    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:39.820684 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.820600    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c2e456-b896-4e5f-a696-7e26fabff35e-istiod-ca-cert" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:39.820943 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.820883    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:39.821732 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.821693    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-kube-api-access-vz6fg" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "kube-api-access-vz6fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:39.822026 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.822003    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-podinfo" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:03:39.822337 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.822303    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-envoy" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:39.822919 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.822900    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-token" pod "65c2e456-b896-4e5f-a696-7e26fabff35e" (UID: "65c2e456-b896-4e5f-a696-7e26fabff35e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:39.921544 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.921520    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:39.921544 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.921542    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz6fg\" (UniqueName: \"kubernetes.io/projected/65c2e456-b896-4e5f-a696-7e26fabff35e-kube-api-access-vz6fg\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:39.921721 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.921552    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:39.921721 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.921560    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:39.921721 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.921569    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c2e456-b896-4e5f-a696-7e26fabff35e-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:39.921721 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:39.921578    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c2e456-b896-4e5f-a696-7e26fabff35e-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:40.669380 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:40.669317    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"]
Nov 02 02:03:40.670714 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:40.670665    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy" containerID="cri-o://20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70" gracePeriod=30
Nov 02 02:03:40.687900 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:40.687863    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4" event={"ID":"65c2e456-b896-4e5f-a696-7e26fabff35e","Type":"ContainerDied","Data":"b04a230eefec660c6e25d7e648b75b07a715fc881c60b163a151baa7638bf127"}
Nov 02 02:03:40.688353 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:40.687924    4287 scope.go:122] "RemoveContainer" containerID="f55faa2402d777cae6f0ade6a7e06f0a3d3377a898cfa561730a805a339b1efa"
Nov 02 02:03:40.688353 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:40.688066    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"
Nov 02 02:03:40.907272 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:40.907218    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"]
Nov 02 02:03:40.911263 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:40.911235    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-269235-openshift-default-6c6ff76bd7-z2qg4"]
Nov 02 02:03:42.237494 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:42.237452    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" path="/var/lib/kubelet/pods/65c2e456-b896-4e5f-a696-7e26fabff35e/volumes"
Nov 02 02:03:43.428576 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:43.428536    4287 patch_prober.go:29] interesting pod/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:03:43.428992 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:43.428591    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:03:45.939914 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:45.939854    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 02:03:46.007080 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007049    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-token\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-token\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007278 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007095    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istiod-ca-cert\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007278 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007125    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-kube-api-access-57cmj\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-kube-api-access-57cmj\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007278 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007242    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-envoy\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007453 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007278    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-podinfo\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007453 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007305    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-credential-socket\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007453 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007341    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-data\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007453 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007410    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-socket\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007687 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007453    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-certs\") pod \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\" (UID: \"7415c203-cde7-40e1-a9a1-b6d3abaf7272\") "
Nov 02 02:03:46.007687 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007629    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-credential-socket" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:46.007687 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007639    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istiod-ca-cert" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:03:46.007880 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007857    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-data" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:46.007973 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.007864    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-socket" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:46.008596 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.008234    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.008596 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.008252    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-certs" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:46.008596 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.008261    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.008596 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.008308    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.008596 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.008323    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.009712 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.009680    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-podinfo" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:03:46.009822 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.009708    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-envoy" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:03:46.010010 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.009981    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-kube-api-access-57cmj" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "kube-api-access-57cmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:46.010086 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.010016    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-token" pod "7415c203-cde7-40e1-a9a1-b6d3abaf7272" (UID: "7415c203-cde7-40e1-a9a1-b6d3abaf7272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:03:46.037689 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:46.037656    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:03:46.109385 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.109279    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.109385 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.109316    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57cmj\" (UniqueName: \"kubernetes.io/projected/7415c203-cde7-40e1-a9a1-b6d3abaf7272-kube-api-access-57cmj\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.109385 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.109333    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.109385 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.109345    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7415c203-cde7-40e1-a9a1-b6d3abaf7272-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.109385 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.109358    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7415c203-cde7-40e1-a9a1-b6d3abaf7272-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:03:46.727150 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.727115    4287 generic.go:356] "Generic (PLEG): container finished" podID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerID="20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70" exitCode=0
Nov 02 02:03:46.727324 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.727179    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"
Nov 02 02:03:46.727324 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.727190    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" event={"ID":"7415c203-cde7-40e1-a9a1-b6d3abaf7272","Type":"ContainerDied","Data":"20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70"}
Nov 02 02:03:46.727324 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.727251    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz" event={"ID":"7415c203-cde7-40e1-a9a1-b6d3abaf7272","Type":"ContainerDied","Data":"b0fad9eec2fbe77304e6710db52137f05c6d31f073ec3839e436a206005cc012"}
Nov 02 02:03:46.727324 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.727277    4287 scope.go:122] "RemoveContainer" containerID="20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70"
Nov 02 02:03:46.737152 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.737132    4287 scope.go:122] "RemoveContainer" containerID="ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce"
Nov 02 02:03:46.747334 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.747315    4287 scope.go:122] "RemoveContainer" containerID="20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70"
Nov 02 02:03:46.747583 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:46.747561    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70\": container with ID starting with 20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70 not found: ID does not exist" containerID="20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70"
Nov 02 02:03:46.747710 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.747593    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70"} err="failed to get container status \"20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70\": rpc error: code = NotFound desc = could not find container \"20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70\": container with ID starting with 20024379adcffcfa27198893048fb55eec2208b72c82d47a4b3fb8d6ba9a8c70 not found: ID does not exist"
Nov 02 02:03:46.747710 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.747637    4287 scope.go:122] "RemoveContainer" containerID="ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce"
Nov 02 02:03:46.747985 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:46.747957    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce\": container with ID starting with ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce not found: ID does not exist" containerID="ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce"
Nov 02 02:03:46.748057 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.747996    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce"} err="failed to get container status \"ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce\": rpc error: code = NotFound desc = could not find container \"ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce\": container with ID starting with ff656067fbe2fb0517e830a1682d0eda6e7460b38e912f7bebe897cddaa0ccce not found: ID does not exist"
Nov 02 02:03:46.891873 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.891836    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"]
Nov 02 02:03:46.941245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:46.941211    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-69c15d-openshift-default-5fb794595-642xz"]
Nov 02 02:03:47.496187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:47.496151    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"]
Nov 02 02:03:48.237228 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:48.237189    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" path="/var/lib/kubelet/pods/7415c203-cde7-40e1-a9a1-b6d3abaf7272/volumes"
Nov 02 02:03:55.361904 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.361866    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"]
Nov 02 02:03:55.362418 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362403    4287 cpu_manager.go:414] "containerMap: removing container" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy"
Nov 02 02:03:55.362480 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362421    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy"
Nov 02 02:03:55.362480 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362436    4287 cpu_manager.go:414] "containerMap: removing container" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main"
Nov 02 02:03:55.362480 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362441    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main"
Nov 02 02:03:55.362480 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362455    4287 cpu_manager.go:414] "containerMap: removing container" podUID="21073d91-773c-4ff8-9486-4b78a3c177a3" containerName="maas-api"
Nov 02 02:03:55.362480 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362464    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21073d91-773c-4ff8-9486-4b78a3c177a3" containerName="maas-api"
Nov 02 02:03:55.362480 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362476    4287 cpu_manager.go:414] "containerMap: removing container" podUID="712e00b8-0705-43e4-ad47-a2e79e00e4fd" containerName="maas-api"
Nov 02 02:03:55.362480 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362481    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712e00b8-0705-43e4-ad47-a2e79e00e4fd" containerName="maas-api"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362488    4287 cpu_manager.go:414] "containerMap: removing container" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362493    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362499    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362503    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362513    4287 cpu_manager.go:414] "containerMap: removing container" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362517    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362526    4287 cpu_manager.go:414] "containerMap: removing container" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362530    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362552    4287 cpu_manager.go:414] "containerMap: removing container" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362557    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362565    4287 cpu_manager.go:414] "containerMap: removing container" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362571    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362581    4287 cpu_manager.go:414] "containerMap: removing container" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362585    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362596    4287 cpu_manager.go:414] "containerMap: removing container" podUID="5df408c8-a8fa-4c65-b307-1fc0c6c43c87" containerName="maas-api"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362601    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df408c8-a8fa-4c65-b307-1fc0c6c43c87" containerName="maas-api"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362623    4287 cpu_manager.go:414] "containerMap: removing container" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362628    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362633    4287 cpu_manager.go:414] "containerMap: removing container" podUID="4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" containerName="maas-api"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362638    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" containerName="maas-api"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362646    4287 cpu_manager.go:414] "containerMap: removing container" podUID="708ed890-4992-4dab-9878-b000a9c3de79" containerName="maas-api"
Nov 02 02:03:55.362703 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362650    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708ed890-4992-4dab-9878-b000a9c3de79" containerName="maas-api"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362720    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="712e00b8-0705-43e4-ad47-a2e79e00e4fd" containerName="maas-api"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362728    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362733    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4390e3-7a12-48cf-ae27-01e87d95d0d6" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362740    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e162987e-acf2-4901-8631-822c903e32d8" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362747    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df408c8-a8fa-4c65-b307-1fc0c6c43c87" containerName="maas-api"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362753    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7862ab77-e085-41b1-b594-44a22aa6b640" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362757    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9088ae-b6ab-4976-9fa0-c77a3bec3ec1" containerName="maas-api"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362763    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1e58e2-df90-44dd-bf75-656c94f4bcdc" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362769    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="708ed890-4992-4dab-9878-b000a9c3de79" containerName="maas-api"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362773    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362778    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8276d1e-3e79-4fa2-8688-14f3da5c3a92" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362783    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362788    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="950028ae-303e-476d-9271-052f2bd8db10" containerName="main"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362793    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c2e456-b896-4e5f-a696-7e26fabff35e" containerName="istio-proxy"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362799    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="21073d91-773c-4ff8-9486-4b78a3c177a3" containerName="maas-api"
Nov 02 02:03:55.363329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.362805    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9364f0-0590-4807-84ff-a579394bf398" containerName="istio-proxy"
Nov 02 02:03:55.367297 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.367283    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.420713 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.420667    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0c954d-serving-cert\""
Nov 02 02:03:55.473184 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.473151    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"]
Nov 02 02:03:55.512431 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.512394    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.512431 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.512433    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60c46d9-232c-4ba5-a854-19eaea3c508d-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.512776 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.512556    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-metrics-certs\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.512776 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.512597    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b926b\" (UniqueName: \"kubernetes.io/projected/e60c46d9-232c-4ba5-a854-19eaea3c508d-kube-api-access-b926b\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.614351 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.614248    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.614351 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.614315    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60c46d9-232c-4ba5-a854-19eaea3c508d-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.614597 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.614376    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-metrics-certs\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.614597 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.614416    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b926b\" (UniqueName: \"kubernetes.io/projected/e60c46d9-232c-4ba5-a854-19eaea3c508d-kube-api-access-b926b\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.614597 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:55.614436    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-serving-cert: secret "maas-api-e2e-api-a-0c954d-serving-cert" not found
Nov 02 02:03:55.614597 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:03:55.614548    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls podName:e60c46d9-232c-4ba5-a854-19eaea3c508d nodeName:}" failed. No retries permitted until 2026-11-02 02:03:56.114525403 +0000 UTC m=+2343.988166467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls") pod "maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9" (UID: "e60c46d9-232c-4ba5-a854-19eaea3c508d") : secret "maas-api-e2e-api-a-0c954d-serving-cert" not found
Nov 02 02:03:55.616161 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.616111    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60c46d9-232c-4ba5-a854-19eaea3c508d-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.618316 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.618283    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-metrics-certs\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:55.675904 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:55.675870    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b926b\" (UniqueName: \"kubernetes.io/projected/e60c46d9-232c-4ba5-a854-19eaea3c508d-kube-api-access-b926b\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:56.121207 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:56.121169    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:56.123866 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:56.123836    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls\") pod \"maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:56.281274 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:56.281244    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:56.658841 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:03:56.658808    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60c46d9_232c_4ba5_a854_19eaea3c508d.slice/crio-daa03de47e8a8458a97954f1d7d12d7aa42cb70b3806dc6ceeee5dc6cefe8d5a WatchSource:0}: Error finding container daa03de47e8a8458a97954f1d7d12d7aa42cb70b3806dc6ceeee5dc6cefe8d5a: Status 404 returned error can't find the container with id daa03de47e8a8458a97954f1d7d12d7aa42cb70b3806dc6ceeee5dc6cefe8d5a
Nov 02 02:03:56.660184 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:56.660164    4287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:03:56.687084 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:56.687049    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"]
Nov 02 02:03:56.785936 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:56.785901    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9" event={"ID":"e60c46d9-232c-4ba5-a854-19eaea3c508d","Type":"ContainerStarted","Data":"daa03de47e8a8458a97954f1d7d12d7aa42cb70b3806dc6ceeee5dc6cefe8d5a"}
Nov 02 02:03:57.795312 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:57.795270    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9" event={"ID":"e60c46d9-232c-4ba5-a854-19eaea3c508d","Type":"ContainerStarted","Data":"403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140"}
Nov 02 02:03:57.795312 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:57.795341    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:03:57.961406 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:03:57.961184    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9" podStartSLOduration=1.951914054 podStartE2EDuration="2.96116905s" totalImagesPullingTime="1.009254996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:57.960559665 +0000 UTC m=+2345.834200716" watchObservedRunningTime="2026-11-02 02:03:57.96116905 +0000 UTC m=+2345.834810102"
Nov 02 02:04:02.517099 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.517034    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main" containerID="cri-o://25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41" gracePeriod=45
Nov 02 02:04:02.802900 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.802876    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 02:04:02.826730 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.826698    4287 generic.go:356] "Generic (PLEG): container finished" podID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerID="25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41" exitCode=0
Nov 02 02:04:02.826852 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.826733    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"
Nov 02 02:04:02.826852 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.826778    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" event={"ID":"94488447-7e8f-4b85-b491-d3b28bf680ed","Type":"ContainerDied","Data":"25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41"}
Nov 02 02:04:02.826852 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.826822    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh" event={"ID":"94488447-7e8f-4b85-b491-d3b28bf680ed","Type":"ContainerDied","Data":"1be66737e60edcaeaefd9280bab921bbed2e901bff82168ed5a5e33e95d924e0"}
Nov 02 02:04:02.826852 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.826841    4287 scope.go:122] "RemoveContainer" containerID="25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41"
Nov 02 02:04:02.836028 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.836009    4287 scope.go:122] "RemoveContainer" containerID="25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41"
Nov 02 02:04:02.836270 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:04:02.836251    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41\": container with ID starting with 25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41 not found: ID does not exist" containerID="25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41"
Nov 02 02:04:02.836309 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.836280    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41"} err="failed to get container status \"25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41\": rpc error: code = NotFound desc = could not find container \"25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41\": container with ID starting with 25a6ccce71abd9f6556d487a966b82860053bc4412162e0bd4b8e3ad4b110b41 not found: ID does not exist"
Nov 02 02:04:02.916581 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.916550    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94488447-7e8f-4b85-b491-d3b28bf680ed-kube-api-access-rrfmq\" (UniqueName: \"kubernetes.io/projected/94488447-7e8f-4b85-b491-d3b28bf680ed-kube-api-access-rrfmq\") pod \"94488447-7e8f-4b85-b491-d3b28bf680ed\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") "
Nov 02 02:04:02.916719 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.916631    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-home\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-home\") pod \"94488447-7e8f-4b85-b491-d3b28bf680ed\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") "
Nov 02 02:04:02.916719 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.916682    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-dshm\") pod \"94488447-7e8f-4b85-b491-d3b28bf680ed\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") "
Nov 02 02:04:02.916824 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.916738    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-model-cache\") pod \"94488447-7e8f-4b85-b491-d3b28bf680ed\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") "
Nov 02 02:04:02.916824 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.916774    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-tmp-dir\") pod \"94488447-7e8f-4b85-b491-d3b28bf680ed\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") "
Nov 02 02:04:02.916824 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.916803    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94488447-7e8f-4b85-b491-d3b28bf680ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/94488447-7e8f-4b85-b491-d3b28bf680ed-tls-certs\") pod \"94488447-7e8f-4b85-b491-d3b28bf680ed\" (UID: \"94488447-7e8f-4b85-b491-d3b28bf680ed\") "
Nov 02 02:04:02.916970 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.916864    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-home" pod "94488447-7e8f-4b85-b491-d3b28bf680ed" (UID: "94488447-7e8f-4b85-b491-d3b28bf680ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:02.917028 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.916980    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-model-cache" pod "94488447-7e8f-4b85-b491-d3b28bf680ed" (UID: "94488447-7e8f-4b85-b491-d3b28bf680ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:02.917028 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.917008    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-tmp-dir" pod "94488447-7e8f-4b85-b491-d3b28bf680ed" (UID: "94488447-7e8f-4b85-b491-d3b28bf680ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:02.917425 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.917335    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:02.917425 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.917364    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:02.917425 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.917379    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:02.919293 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.919232    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-dshm" pod "94488447-7e8f-4b85-b491-d3b28bf680ed" (UID: "94488447-7e8f-4b85-b491-d3b28bf680ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:02.920054 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.919278    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94488447-7e8f-4b85-b491-d3b28bf680ed-kube-api-access-rrfmq" pod "94488447-7e8f-4b85-b491-d3b28bf680ed" (UID: "94488447-7e8f-4b85-b491-d3b28bf680ed"). InnerVolumeSpecName "kube-api-access-rrfmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:02.920054 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:02.919760    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94488447-7e8f-4b85-b491-d3b28bf680ed-tls-certs" pod "94488447-7e8f-4b85-b491-d3b28bf680ed" (UID: "94488447-7e8f-4b85-b491-d3b28bf680ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:04:03.019324 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:03.019288    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrfmq\" (UniqueName: \"kubernetes.io/projected/94488447-7e8f-4b85-b491-d3b28bf680ed-kube-api-access-rrfmq\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:03.019324 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:03.019327    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94488447-7e8f-4b85-b491-d3b28bf680ed-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:03.019518 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:03.019343    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94488447-7e8f-4b85-b491-d3b28bf680ed-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:03.253439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:03.253402    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"]
Nov 02 02:04:03.256943 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:03.256913    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/distinct-2-w1-f276c3-kserve-74c765996d-rl9dh"]
Nov 02 02:04:03.806671 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:03.806642    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:04:04.238446 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:04.238367    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" path="/var/lib/kubelet/pods/94488447-7e8f-4b85-b491-d3b28bf680ed/volumes"
Nov 02 02:04:16.053119 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:04:16.053080    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:04:21.849989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:21.849952    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"]
Nov 02 02:04:21.850717 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:21.850196    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj" podUID="048d59f4-304c-4e3f-a645-05475489a0da" containerName="maas-api" containerID="cri-o://4e682b6cce0479b09b64f227491f15a8f3f0bdb16b11d52c945b28181986e087" gracePeriod=30
Nov 02 02:04:22.006953 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.005923    4287 generic.go:356] "Generic (PLEG): container finished" podID="048d59f4-304c-4e3f-a645-05475489a0da" containerID="4e682b6cce0479b09b64f227491f15a8f3f0bdb16b11d52c945b28181986e087" exitCode=0
Nov 02 02:04:22.006953 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.006112    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj" event={"ID":"048d59f4-304c-4e3f-a645-05475489a0da","Type":"ContainerDied","Data":"4e682b6cce0479b09b64f227491f15a8f3f0bdb16b11d52c945b28181986e087"}
Nov 02 02:04:22.161410 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.161377    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 02:04:22.236007 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.235346    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048d59f4-304c-4e3f-a645-05475489a0da-kube-api-access-hwxd4\" (UniqueName: \"kubernetes.io/projected/048d59f4-304c-4e3f-a645-05475489a0da-kube-api-access-hwxd4\") pod \"048d59f4-304c-4e3f-a645-05475489a0da\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") "
Nov 02 02:04:22.236007 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.235457    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-metrics-certs\") pod \"048d59f4-304c-4e3f-a645-05475489a0da\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") "
Nov 02 02:04:22.236007 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.235546    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048d59f4-304c-4e3f-a645-05475489a0da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048d59f4-304c-4e3f-a645-05475489a0da-openshift-service-ca\") pod \"048d59f4-304c-4e3f-a645-05475489a0da\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") "
Nov 02 02:04:22.236007 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.235565    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls\") pod \"048d59f4-304c-4e3f-a645-05475489a0da\" (UID: \"048d59f4-304c-4e3f-a645-05475489a0da\") "
Nov 02 02:04:22.236007 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.235992    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048d59f4-304c-4e3f-a645-05475489a0da-openshift-service-ca" pod "048d59f4-304c-4e3f-a645-05475489a0da" (UID: "048d59f4-304c-4e3f-a645-05475489a0da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:22.238777 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.238744    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-metrics-certs" pod "048d59f4-304c-4e3f-a645-05475489a0da" (UID: "048d59f4-304c-4e3f-a645-05475489a0da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:04:22.238914 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.238744    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048d59f4-304c-4e3f-a645-05475489a0da-kube-api-access-hwxd4" pod "048d59f4-304c-4e3f-a645-05475489a0da" (UID: "048d59f4-304c-4e3f-a645-05475489a0da"). InnerVolumeSpecName "kube-api-access-hwxd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:22.240374 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.240350    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls" pod "048d59f4-304c-4e3f-a645-05475489a0da" (UID: "048d59f4-304c-4e3f-a645-05475489a0da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:04:22.338246 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.338210    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwxd4\" (UniqueName: \"kubernetes.io/projected/048d59f4-304c-4e3f-a645-05475489a0da-kube-api-access-hwxd4\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:22.338445 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.338254    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:22.338445 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.338269    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048d59f4-304c-4e3f-a645-05475489a0da-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:22.338445 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:22.338283    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048d59f4-304c-4e3f-a645-05475489a0da-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:23.013018 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:23.012987    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"
Nov 02 02:04:23.013018 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:23.013007    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj" event={"ID":"048d59f4-304c-4e3f-a645-05475489a0da","Type":"ContainerDied","Data":"d26c29977c5e255db8e0a2f4df9fec88dfc8addbb2564039879206ebd220b4dd"}
Nov 02 02:04:23.013574 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:23.013089    4287 scope.go:122] "RemoveContainer" containerID="4e682b6cce0479b09b64f227491f15a8f3f0bdb16b11d52c945b28181986e087"
Nov 02 02:04:23.176776 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:23.176745    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"]
Nov 02 02:04:23.226586 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:23.226557    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-187478-845f545876-cvqhj"]
Nov 02 02:04:24.257708 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:24.257670    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048d59f4-304c-4e3f-a645-05475489a0da" path="/var/lib/kubelet/pods/048d59f4-304c-4e3f-a645-05475489a0da/volumes"
Nov 02 02:04:31.031594 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:31.031560    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"]
Nov 02 02:04:31.032000 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:31.031828    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy" containerID="cri-o://6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec" gracePeriod=30
Nov 02 02:04:36.308470 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.308408    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 02:04:36.394978 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.394949    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-envoy\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395013    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-credential-socket\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395035    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb550fa5-5b81-4547-82c3-8fd91623df50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb550fa5-5b81-4547-82c3-8fd91623df50-istiod-ca-cert\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395075    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-token\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-token\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395109    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-data\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395347 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395176    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-podinfo\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395347 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395234    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-kube-api-access-gm66g\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-kube-api-access-gm66g\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395347 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395295    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-credential-socket" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:36.395347 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395316    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-socket\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395567 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395366    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-certs\") pod \"fb550fa5-5b81-4547-82c3-8fd91623df50\" (UID: \"fb550fa5-5b81-4547-82c3-8fd91623df50\") "
Nov 02 02:04:36.395567 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395525    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-socket" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:36.395567 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395533    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb550fa5-5b81-4547-82c3-8fd91623df50-istiod-ca-cert" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:04:36.395567 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.395544    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-data" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:36.396066 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.396045    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:36.396103 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.396074    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:36.396103 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.396092    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:36.396179 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.396107    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb550fa5-5b81-4547-82c3-8fd91623df50-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:36.396343 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.396323    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-certs" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:36.397507 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.397483    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-envoy" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:04:36.397507 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.397499    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-token" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:36.398029 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.398004    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-kube-api-access-gm66g" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "kube-api-access-gm66g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:04:36.398188 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.398170    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-podinfo" pod "fb550fa5-5b81-4547-82c3-8fd91623df50" (UID: "fb550fa5-5b81-4547-82c3-8fd91623df50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:04:36.497232 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.497199    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:36.497346 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.497234    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm66g\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-kube-api-access-gm66g\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:36.497346 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.497251    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:36.497346 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.497266    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:36.497346 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:36.497280    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb550fa5-5b81-4547-82c3-8fd91623df50-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:04:37.100627 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.100585    4287 generic.go:356] "Generic (PLEG): container finished" podID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerID="6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec" exitCode=0
Nov 02 02:04:37.100807 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.100648    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" event={"ID":"fb550fa5-5b81-4547-82c3-8fd91623df50","Type":"ContainerDied","Data":"6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec"}
Nov 02 02:04:37.100807 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.100675    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"
Nov 02 02:04:37.100807 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.100708    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp" event={"ID":"fb550fa5-5b81-4547-82c3-8fd91623df50","Type":"ContainerDied","Data":"11bfbf7f93bd6d799e8933c64952e604c12606baac11f12da2edd7496fe18507"}
Nov 02 02:04:37.100807 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.100730    4287 scope.go:122] "RemoveContainer" containerID="6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec"
Nov 02 02:04:37.110930 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.110915    4287 scope.go:122] "RemoveContainer" containerID="6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec"
Nov 02 02:04:37.111192 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:04:37.111166    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec\": container with ID starting with 6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec not found: ID does not exist" containerID="6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec"
Nov 02 02:04:37.111245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.111201    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec"} err="failed to get container status \"6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec\": rpc error: code = NotFound desc = could not find container \"6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec\": container with ID starting with 6afe06985a3b1ef0da74494a6d978d4a31e80f63e884dda85c14876d3ffc5cec not found: ID does not exist"
Nov 02 02:04:37.317314 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.317275    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"]
Nov 02 02:04:37.365726 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:37.365658    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-2x9bp"]
Nov 02 02:04:38.239669 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:38.239631    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" path="/var/lib/kubelet/pods/fb550fa5-5b81-4547-82c3-8fd91623df50/volumes"
Nov 02 02:04:46.040984 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:04:46.040920    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:04:52.573760 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:52.573733    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:04:52.595389 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:04:52.595365    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:05:16.038044 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:05:16.038000    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:05:41.017903 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.017868    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"]
Nov 02 02:05:41.018420 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.018124    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp" podUID="966d6023-b96f-4e93-8b05-f849e473b4fe" containerName="maas-api" containerID="cri-o://86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e" gracePeriod=30
Nov 02 02:05:41.286069 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.286046    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 02:05:41.415146 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.415106    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966d6023-b96f-4e93-8b05-f849e473b4fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966d6023-b96f-4e93-8b05-f849e473b4fe-openshift-service-ca\") pod \"966d6023-b96f-4e93-8b05-f849e473b4fe\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") "
Nov 02 02:05:41.417847 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.415424    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-metrics-certs\") pod \"966d6023-b96f-4e93-8b05-f849e473b4fe\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") "
Nov 02 02:05:41.417847 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.415578    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966d6023-b96f-4e93-8b05-f849e473b4fe-kube-api-access-hhw67\" (UniqueName: \"kubernetes.io/projected/966d6023-b96f-4e93-8b05-f849e473b4fe-kube-api-access-hhw67\") pod \"966d6023-b96f-4e93-8b05-f849e473b4fe\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") "
Nov 02 02:05:41.417847 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.415639    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls\") pod \"966d6023-b96f-4e93-8b05-f849e473b4fe\" (UID: \"966d6023-b96f-4e93-8b05-f849e473b4fe\") "
Nov 02 02:05:41.418308 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.418274    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966d6023-b96f-4e93-8b05-f849e473b4fe-openshift-service-ca" pod "966d6023-b96f-4e93-8b05-f849e473b4fe" (UID: "966d6023-b96f-4e93-8b05-f849e473b4fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:41.424027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.424003    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls" pod "966d6023-b96f-4e93-8b05-f849e473b4fe" (UID: "966d6023-b96f-4e93-8b05-f849e473b4fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:05:41.424281 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.424264    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-metrics-certs" pod "966d6023-b96f-4e93-8b05-f849e473b4fe" (UID: "966d6023-b96f-4e93-8b05-f849e473b4fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:05:41.431619 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.430766    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966d6023-b96f-4e93-8b05-f849e473b4fe-kube-api-access-hhw67" pod "966d6023-b96f-4e93-8b05-f849e473b4fe" (UID: "966d6023-b96f-4e93-8b05-f849e473b4fe"). InnerVolumeSpecName "kube-api-access-hhw67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:41.479056 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.479023    4287 generic.go:356] "Generic (PLEG): container finished" podID="966d6023-b96f-4e93-8b05-f849e473b4fe" containerID="86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e" exitCode=0
Nov 02 02:05:41.479231 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.479111    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"
Nov 02 02:05:41.479231 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.479118    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp" event={"ID":"966d6023-b96f-4e93-8b05-f849e473b4fe","Type":"ContainerDied","Data":"86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e"}
Nov 02 02:05:41.479231 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.479177    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp" event={"ID":"966d6023-b96f-4e93-8b05-f849e473b4fe","Type":"ContainerDied","Data":"7b4ebf77203518f46e088ffc3d479a18913c0656408e3c271505d81769f62c3c"}
Nov 02 02:05:41.479231 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.479206    4287 scope.go:122] "RemoveContainer" containerID="86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e"
Nov 02 02:05:41.489453 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.489329    4287 scope.go:122] "RemoveContainer" containerID="86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e"
Nov 02 02:05:41.489737 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:05:41.489704    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e\": container with ID starting with 86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e not found: ID does not exist" containerID="86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e"
Nov 02 02:05:41.489835 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.489743    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e"} err="failed to get container status \"86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e\": rpc error: code = NotFound desc = could not find container \"86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e\": container with ID starting with 86de3955974e68300423d46c6b1c4c17c1527cdc2560edd8e1921df4ed33741e not found: ID does not exist"
Nov 02 02:05:41.517654 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.517600    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhw67\" (UniqueName: \"kubernetes.io/projected/966d6023-b96f-4e93-8b05-f849e473b4fe-kube-api-access-hhw67\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:41.517786 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.517661    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:41.517786 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.517679    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/966d6023-b96f-4e93-8b05-f849e473b4fe-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:41.517786 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.517694    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966d6023-b96f-4e93-8b05-f849e473b4fe-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:41.644087 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.643997    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"]
Nov 02 02:05:41.651909 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:41.651877    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-f276c3-b4bb57f99-w8nnp"]
Nov 02 02:05:42.237128 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:42.237094    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966d6023-b96f-4e93-8b05-f849e473b4fe" path="/var/lib/kubelet/pods/966d6023-b96f-4e93-8b05-f849e473b4fe/volumes"
Nov 02 02:05:45.967757 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:45.967721    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"]
Nov 02 02:05:45.968171 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:45.967952    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy" containerID="cri-o://521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce" gracePeriod=30
Nov 02 02:05:46.038949 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:05:46.038905    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:05:47.880648 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.880616    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"]
Nov 02 02:05:47.881056 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881044    4287 cpu_manager.go:414] "containerMap: removing container" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy"
Nov 02 02:05:47.881101 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881058    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7415c203-cde7-40e1-a9a1-b6d3abaf7272" containerName="istio-proxy"
Nov 02 02:05:47.881101 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881075    4287 cpu_manager.go:414] "containerMap: removing container" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main"
Nov 02 02:05:47.881101 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881081    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main"
Nov 02 02:05:47.881101 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881091    4287 cpu_manager.go:414] "containerMap: removing container" podUID="966d6023-b96f-4e93-8b05-f849e473b4fe" containerName="maas-api"
Nov 02 02:05:47.881101 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881096    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966d6023-b96f-4e93-8b05-f849e473b4fe" containerName="maas-api"
Nov 02 02:05:47.881251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881107    4287 cpu_manager.go:414] "containerMap: removing container" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy"
Nov 02 02:05:47.881251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881111    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy"
Nov 02 02:05:47.881251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881124    4287 cpu_manager.go:414] "containerMap: removing container" podUID="048d59f4-304c-4e3f-a645-05475489a0da" containerName="maas-api"
Nov 02 02:05:47.881251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881129    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048d59f4-304c-4e3f-a645-05475489a0da" containerName="maas-api"
Nov 02 02:05:47.881251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881194    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb550fa5-5b81-4547-82c3-8fd91623df50" containerName="istio-proxy"
Nov 02 02:05:47.881251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881205    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="048d59f4-304c-4e3f-a645-05475489a0da" containerName="maas-api"
Nov 02 02:05:47.881251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881213    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="94488447-7e8f-4b85-b491-d3b28bf680ed" containerName="main"
Nov 02 02:05:47.881251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.881221    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="966d6023-b96f-4e93-8b05-f849e473b4fe" containerName="maas-api"
Nov 02 02:05:47.886056 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.886035    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.938427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.938393    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9039bf-openshift-default-dockercfg-bkcdd\""
Nov 02 02:05:47.991463 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.991427    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"]
Nov 02 02:05:47.996134 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996101    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-certs\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.996263 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996170    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-credential-socket\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.996309 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996270    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-podinfo\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.996382 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996365    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-envoy\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.996487 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996465    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-socket\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.996523 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996502    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-data\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.996602 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996583    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shn68\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-kube-api-access-shn68\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.996716 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996694    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-token\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:47.996826 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:47.996807    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589391c-05a1-4ccf-b816-9615d5a6d69f-istiod-ca-cert\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.097517 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097468    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-certs\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.097755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097537    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-credential-socket\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.097755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097566    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-podinfo\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.097755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097663    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-envoy\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.097755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097692    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-socket\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.097755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097714    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-data\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.097755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097747    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shn68\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-kube-api-access-shn68\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.098086 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097790    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-token\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.098086 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097855    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589391c-05a1-4ccf-b816-9615d5a6d69f-istiod-ca-cert\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.098086 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.097967    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-certs\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.098243 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.098168    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-socket\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.098305 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.098258    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-data\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.098630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.098573    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-credential-socket\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.098726 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.098635    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589391c-05a1-4ccf-b816-9615d5a6d69f-istiod-ca-cert\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.100337 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.100315    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-podinfo\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.100521 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.100494    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-envoy\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.162666 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.162588    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-token\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.162852 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.162830    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shn68\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-kube-api-access-shn68\") pod \"e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") " pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.198988 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.198955    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:48.642054 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.642026    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"]
Nov 02 02:05:48.643456 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:05:48.643425    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0589391c_05a1_4ccf_b816_9615d5a6d69f.slice/crio-ab90e139000b05e81f45affa24128c7ae391db182ff52f914a9e03118f8cd5d0 WatchSource:0}: Error finding container ab90e139000b05e81f45affa24128c7ae391db182ff52f914a9e03118f8cd5d0: Status 404 returned error can't find the container with id ab90e139000b05e81f45affa24128c7ae391db182ff52f914a9e03118f8cd5d0
Nov 02 02:05:48.645888 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.645854    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:05:48.645992 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.645912    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:05:48.645992 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:48.645940    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:05:49.527546 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:49.527514    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" event={"ID":"0589391c-05a1-4ccf-b816-9615d5a6d69f","Type":"ContainerStarted","Data":"ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5"}
Nov 02 02:05:49.528044 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:49.527565    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" event={"ID":"0589391c-05a1-4ccf-b816-9615d5a6d69f","Type":"ContainerStarted","Data":"ab90e139000b05e81f45affa24128c7ae391db182ff52f914a9e03118f8cd5d0"}
Nov 02 02:05:49.701413 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:49.701359    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" podStartSLOduration=2.70134595 podStartE2EDuration="2.70134595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:05:49.698441583 +0000 UTC m=+2457.572082637" watchObservedRunningTime="2026-11-02 02:05:49.70134595 +0000 UTC m=+2457.574987002"
Nov 02 02:05:50.199796 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:50.199755    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:50.205431 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:50.205405    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:50.533307 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:50.533276    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:50.534340 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:50.534309    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:05:50.868200 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:50.868122    4287 patch_prober.go:29] interesting pod/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 02:05:50.868200 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:50.868180    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 02:05:51.229741 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.229718    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 02:05:51.335807 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.335767    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-certs\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336002 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.335843    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istiod-ca-cert\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336002 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.335874    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-podinfo\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336002 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.335974    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-kube-api-access-rbfgw\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-kube-api-access-rbfgw\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336175 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336002    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-envoy\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336175 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336065    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-data\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336175 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336050    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-certs" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:51.336175 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336127    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-token\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-token\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336390 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336207    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-socket\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336390 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336227    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istiod-ca-cert" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:05:51.336390 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336252    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-credential-socket\") pod \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\" (UID: \"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db\") "
Nov 02 02:05:51.336577 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336547    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-credential-socket" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:51.336577 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336557    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-socket" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:51.336902 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.336858    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-data" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:51.337333 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.337309    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.337437 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.337339    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.337437 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.337357    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.337437 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.337373    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.337437 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.337390    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.338529 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.338508    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-podinfo" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:05:51.338529 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.338520    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-kube-api-access-rbfgw" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "kube-api-access-rbfgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:51.338696 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.338520    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-token" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:05:51.338696 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.338643    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-envoy" pod "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" (UID: "bdd592bd-25cf-4ae9-9cb2-ab2ba12335db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:05:51.439341 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.439184    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.439341 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.439230    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbfgw\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-kube-api-access-rbfgw\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.439341 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.439248    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.439341 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.439262    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:05:51.567329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.567288    4287 generic.go:356] "Generic (PLEG): container finished" podID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerID="521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce" exitCode=0
Nov 02 02:05:51.567874 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.567354    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"
Nov 02 02:05:51.567874 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.567468    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" event={"ID":"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db","Type":"ContainerDied","Data":"521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce"}
Nov 02 02:05:51.567874 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.567513    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p" event={"ID":"bdd592bd-25cf-4ae9-9cb2-ab2ba12335db","Type":"ContainerDied","Data":"bbdf46d913101a1e874a1787d5e6bcdb46fe039aa623e1e85d668c12d784129f"}
Nov 02 02:05:51.567874 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.567540    4287 scope.go:122] "RemoveContainer" containerID="521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce"
Nov 02 02:05:51.584904 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.584875    4287 scope.go:122] "RemoveContainer" containerID="60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9"
Nov 02 02:05:51.595450 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.595426    4287 scope.go:122] "RemoveContainer" containerID="521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce"
Nov 02 02:05:51.595837 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:05:51.595809    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce\": container with ID starting with 521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce not found: ID does not exist" containerID="521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce"
Nov 02 02:05:51.595891 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.595851    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce"} err="failed to get container status \"521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce\": rpc error: code = NotFound desc = could not find container \"521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce\": container with ID starting with 521eed12288e74f45272d1b4508341113a364ca16cf37ced5c911e7efb8456ce not found: ID does not exist"
Nov 02 02:05:51.595891 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.595879    4287 scope.go:122] "RemoveContainer" containerID="60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9"
Nov 02 02:05:51.596238 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:05:51.596217    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9\": container with ID starting with 60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9 not found: ID does not exist" containerID="60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9"
Nov 02 02:05:51.596293 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.596246    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9"} err="failed to get container status \"60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9\": rpc error: code = NotFound desc = could not find container \"60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9\": container with ID starting with 60613e34e13ec5d91560ae67565b43c420f76b065ef6922f4dafcb2c2868b3e9 not found: ID does not exist"
Nov 02 02:05:51.737159 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.737068    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"]
Nov 02 02:05:51.745229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:51.745199    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-07ac70d0-openshift-default-fb5f557bf-q6r9p"]
Nov 02 02:05:52.239132 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:52.239102    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" path="/var/lib/kubelet/pods/bdd592bd-25cf-4ae9-9cb2-ab2ba12335db/volumes"
Nov 02 02:05:58.141133 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.141099    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"]
Nov 02 02:05:58.141755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.141735    4287 cpu_manager.go:414] "containerMap: removing container" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy"
Nov 02 02:05:58.141853 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.141756    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy"
Nov 02 02:05:58.141917 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.141891    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy"
Nov 02 02:05:58.141917 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.141905    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy"
Nov 02 02:05:58.146637 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.146588    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.201022 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.201001    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c78625-serving-cert\""
Nov 02 02:05:58.253287 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.253256    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"]
Nov 02 02:05:58.323537 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.323508    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2a55-b4ef-4548-87e5-03df6bc720e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.323707 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.323548    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-maas-api-tls\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.323759 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.323728    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-metrics-certs\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.323817 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.323804    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5tvn\" (UniqueName: \"kubernetes.io/projected/d32a2a55-b4ef-4548-87e5-03df6bc720e6-kube-api-access-z5tvn\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.425193 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.425101    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-metrics-certs\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.425193 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.425186    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tvn\" (UniqueName: \"kubernetes.io/projected/d32a2a55-b4ef-4548-87e5-03df6bc720e6-kube-api-access-z5tvn\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.425431 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.425225    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2a55-b4ef-4548-87e5-03df6bc720e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.425431 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.425246    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-maas-api-tls\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.426051 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.425961    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2a55-b4ef-4548-87e5-03df6bc720e6-openshift-service-ca\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.427807 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.427783    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-maas-api-tls\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.427932 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.427909    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-metrics-certs\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.486327 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.486288    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tvn\" (UniqueName: \"kubernetes.io/projected/d32a2a55-b4ef-4548-87e5-03df6bc720e6-kube-api-access-z5tvn\") pod \"maas-api-e2e-api-b-c78625-6459476c9-qj9ft\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:58.759316 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:58.759243    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:05:59.133394 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:59.133370    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"]
Nov 02 02:05:59.134808 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:05:59.134782    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32a2a55_b4ef_4548_87e5_03df6bc720e6.slice/crio-118b477a3bd7b2b276df17db1b8535725dd864ddfc91d5e3b1392b4de77d5eb0 WatchSource:0}: Error finding container 118b477a3bd7b2b276df17db1b8535725dd864ddfc91d5e3b1392b4de77d5eb0: Status 404 returned error can't find the container with id 118b477a3bd7b2b276df17db1b8535725dd864ddfc91d5e3b1392b4de77d5eb0
Nov 02 02:05:59.612790 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:05:59.612743    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft" event={"ID":"d32a2a55-b4ef-4548-87e5-03df6bc720e6","Type":"ContainerStarted","Data":"118b477a3bd7b2b276df17db1b8535725dd864ddfc91d5e3b1392b4de77d5eb0"}
Nov 02 02:06:00.619371 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:00.619337    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft" event={"ID":"d32a2a55-b4ef-4548-87e5-03df6bc720e6","Type":"ContainerStarted","Data":"5228f3c0ff56d46ea66dbe5f36fd55839a6216f85428286e1d4b97e66fb68864"}
Nov 02 02:06:00.619791 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:00.619471    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:06:00.736891 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:00.736835    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft" podStartSLOduration=1.901892682 podStartE2EDuration="2.73681771s" totalImagesPullingTime="834.925028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:00.736156898 +0000 UTC m=+2468.609797951" watchObservedRunningTime="2026-11-02 02:06:00.73681771 +0000 UTC m=+2468.610458767"
Nov 02 02:06:06.630167 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:06.630085    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:06:16.038780 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:06:16.038739    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:06:35.156261 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.156229    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"]
Nov 02 02:06:35.156724 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.156711    4287 cpu_manager.go:414] "containerMap: removing container" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy"
Nov 02 02:06:35.156777 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.156725    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd592bd-25cf-4ae9-9cb2-ab2ba12335db" containerName="istio-proxy"
Nov 02 02:06:35.160282 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.160267    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214319 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214279    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214483 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214330    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214483 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214347    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bd95e2-4708-48e4-831d-05c676377146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214483 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214455    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214629 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214498    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214669 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214640    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214716 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214702    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bd95e2-4708-48e4-831d-05c676377146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214754 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214725    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjb9\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-kube-api-access-xrjb9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.214819 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.214807    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.315701 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.315666    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bd95e2-4708-48e4-831d-05c676377146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.315884 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.315708    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjb9\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-kube-api-access-xrjb9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.315884 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.315775    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.316024 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.315950    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.316024 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.316001    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.316128 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.316044    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bd95e2-4708-48e4-831d-05c676377146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.316128 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.316093    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.316231 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.316137    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.316231 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.316203    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.316658 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.316600    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.316807 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.316783    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bd95e2-4708-48e4-831d-05c676377146-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.317036 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.317010    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.317212 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.317087    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.317381 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.317343    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.319216 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.319194    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.319461 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.319437    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bd95e2-4708-48e4-831d-05c676377146-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.360725 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.360692    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"]
Nov 02 02:06:35.377358 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.377329    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.377510 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.377329    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjb9\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-kube-api-access-xrjb9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxwdk\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.473293 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.473195    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:35.905821 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.905793    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"]
Nov 02 02:06:35.906658 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:06:35.906630    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bd95e2_4708_48e4_831d_05c676377146.slice/crio-05827279d0e1e8ec0bf0ee491fe7d564c681b2c24ae4851b5a88b4e202940a10 WatchSource:0}: Error finding container 05827279d0e1e8ec0bf0ee491fe7d564c681b2c24ae4851b5a88b4e202940a10: Status 404 returned error can't find the container with id 05827279d0e1e8ec0bf0ee491fe7d564c681b2c24ae4851b5a88b4e202940a10
Nov 02 02:06:35.909064 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.909033    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:06:35.909124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.909095    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:06:35.909163 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:35.909124    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:06:36.852868 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:36.852833    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" event={"ID":"b6bd95e2-4708-48e4-831d-05c676377146","Type":"ContainerStarted","Data":"add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf"}
Nov 02 02:06:36.853353 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:36.852885    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" event={"ID":"b6bd95e2-4708-48e4-831d-05c676377146","Type":"ContainerStarted","Data":"05827279d0e1e8ec0bf0ee491fe7d564c681b2c24ae4851b5a88b4e202940a10"}
Nov 02 02:06:37.029911 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:37.029846    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podStartSLOduration=2.029830293 podStartE2EDuration="2.029830293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:37.02310869 +0000 UTC m=+2504.896749756" watchObservedRunningTime="2026-11-02 02:06:37.029830293 +0000 UTC m=+2504.903471422"
Nov 02 02:06:37.473840 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:37.473797    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:37.475729 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:37.475704    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 02 02:06:37.475802 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:37.475775    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 02 02:06:38.474115 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:38.474076    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 02 02:06:38.474683 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:38.474154    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 02 02:06:39.474193 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:39.474027    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 02 02:06:39.474193 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:39.474099    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 02 02:06:40.474037 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:40.473993    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 02 02:06:40.474213 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:40.474061    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 02 02:06:41.473949 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:41.473907    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 02 02:06:41.474129 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:41.473972    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 02 02:06:42.473715 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:42.473678    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 02 02:06:42.474134 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:42.473733    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 02 02:06:43.474213 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:43.474173    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 02 02:06:43.474655 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:43.474240    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 02 02:06:44.929434 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:44.929404    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:44.929948 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:44.929818    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:44.931059 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:44.931039    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:06:46.038996 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:06:46.038960    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:06:51.622122 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.622091    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"]
Nov 02 02:06:51.629816 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.629781    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.682260 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.682232    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9039bf-serving-cert\""
Nov 02 02:06:51.684417 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.684381    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"]
Nov 02 02:06:51.731245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.731213    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-metrics-certs\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.731397 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.731374    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkmk7\" (UniqueName: \"kubernetes.io/projected/4b496963-737d-4d4b-b618-ad31f1305896-kube-api-access-qkmk7\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.731441 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.731413    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b496963-737d-4d4b-b618-ad31f1305896-openshift-service-ca\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.731483 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.731438    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.832124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.832090    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-metrics-certs\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.832487 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.832250    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkmk7\" (UniqueName: \"kubernetes.io/projected/4b496963-737d-4d4b-b618-ad31f1305896-kube-api-access-qkmk7\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.832487 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.832292    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b496963-737d-4d4b-b618-ad31f1305896-openshift-service-ca\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.832487 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.832320    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.832751 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:06:51.832508    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-serving-cert: secret "maas-api-e2e-api-b-9039bf-serving-cert" not found
Nov 02 02:06:51.832751 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:06:51.832580    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls podName:4b496963-737d-4d4b-b618-ad31f1305896 nodeName:}" failed. No retries permitted until 2026-11-02 02:06:52.332563951 +0000 UTC m=+2520.206204995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls") pod "maas-api-e2e-api-b-9039bf-56d8557bff-mndk6" (UID: "4b496963-737d-4d4b-b618-ad31f1305896") : secret "maas-api-e2e-api-b-9039bf-serving-cert" not found
Nov 02 02:06:51.833073 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.833049    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b496963-737d-4d4b-b618-ad31f1305896-openshift-service-ca\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.834546 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.834527    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-metrics-certs\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:51.895899 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:51.895831    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkmk7\" (UniqueName: \"kubernetes.io/projected/4b496963-737d-4d4b-b618-ad31f1305896-kube-api-access-qkmk7\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:52.337869 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:52.337833    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:52.340170 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:52.340145    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls\") pod \"maas-api-e2e-api-b-9039bf-56d8557bff-mndk6\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:52.546401 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:52.546374    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:52.845825 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:52.845797    4287 scope.go:122] "RemoveContainer" containerID="a235768bb8224810b8fd7bac90fdd3dd501fead529bd06996f038f249440baee"
Nov 02 02:06:52.925371 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:52.925347    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"]
Nov 02 02:06:52.926707 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:06:52.926681    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b496963_737d_4d4b_b618_ad31f1305896.slice/crio-f20dfb50aeac6bbe92c52af5e377893f394dcd6e1608d6718cb385b9891d70fb WatchSource:0}: Error finding container f20dfb50aeac6bbe92c52af5e377893f394dcd6e1608d6718cb385b9891d70fb: Status 404 returned error can't find the container with id f20dfb50aeac6bbe92c52af5e377893f394dcd6e1608d6718cb385b9891d70fb
Nov 02 02:06:52.970853 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:52.970821    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6" event={"ID":"4b496963-737d-4d4b-b618-ad31f1305896","Type":"ContainerStarted","Data":"f20dfb50aeac6bbe92c52af5e377893f394dcd6e1608d6718cb385b9891d70fb"}
Nov 02 02:06:53.978670 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:53.978631    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6" event={"ID":"4b496963-737d-4d4b-b618-ad31f1305896","Type":"ContainerStarted","Data":"0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f"}
Nov 02 02:06:53.979096 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:53.978698    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:06:54.098585 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:54.098537    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6" podStartSLOduration=2.295486907 podStartE2EDuration="3.098523727s" totalImagesPullingTime="803.03682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:54.095410382 +0000 UTC m=+2521.969051446" watchObservedRunningTime="2026-11-02 02:06:54.098523727 +0000 UTC m=+2521.972164779"
Nov 02 02:06:59.989598 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:06:59.989564    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:07:10.428187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:10.428155    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"]
Nov 02 02:07:10.477054 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:10.477020    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"]
Nov 02 02:07:16.039862 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:07:16.039826    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:07:25.449276 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.449232    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" containerID="cri-o://b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709" gracePeriod=45
Nov 02 02:07:25.497427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.497396    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main" containerID="cri-o://9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098" gracePeriod=45
Nov 02 02:07:25.630652 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.627242    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused"
Nov 02 02:07:25.758908 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.758880    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 02:07:25.787880 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.787856    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 02:07:25.824393 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824369    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-tmp-dir\") pod \"19694632-5293-4f90-bd57-e118aa208699\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") "
Nov 02 02:07:25.824542 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824468    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19694632-5293-4f90-bd57-e118aa208699-tls-certs\" (UniqueName: \"kubernetes.io/secret/19694632-5293-4f90-bd57-e118aa208699-tls-certs\") pod \"19694632-5293-4f90-bd57-e118aa208699\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") "
Nov 02 02:07:25.824542 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824516    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-model-cache\") pod \"19694632-5293-4f90-bd57-e118aa208699\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") "
Nov 02 02:07:25.824671 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824544    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-home\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-home\") pod \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") "
Nov 02 02:07:25.824671 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824659    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-home\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-home\") pod \"19694632-5293-4f90-bd57-e118aa208699\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") "
Nov 02 02:07:25.824785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824706    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-model-cache\") pod \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") "
Nov 02 02:07:25.824785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824655    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-tmp-dir" pod "19694632-5293-4f90-bd57-e118aa208699" (UID: "19694632-5293-4f90-bd57-e118aa208699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.824785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824739    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b097b84-fb8c-41f6-8b74-9f9a29245b59-kube-api-access-wxbvj\" (UniqueName: \"kubernetes.io/projected/9b097b84-fb8c-41f6-8b74-9f9a29245b59-kube-api-access-wxbvj\") pod \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") "
Nov 02 02:07:25.824785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824763    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-model-cache" pod "19694632-5293-4f90-bd57-e118aa208699" (UID: "19694632-5293-4f90-bd57-e118aa208699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.824989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824771    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-dshm\") pod \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") "
Nov 02 02:07:25.824989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824857    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-dshm\") pod \"19694632-5293-4f90-bd57-e118aa208699\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") "
Nov 02 02:07:25.824989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824864    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-home" pod "9b097b84-fb8c-41f6-8b74-9f9a29245b59" (UID: "9b097b84-fb8c-41f6-8b74-9f9a29245b59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.824989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824918    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tmp-dir\") pod \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") "
Nov 02 02:07:25.824989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824936    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-model-cache" pod "9b097b84-fb8c-41f6-8b74-9f9a29245b59" (UID: "9b097b84-fb8c-41f6-8b74-9f9a29245b59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.824989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.824962    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tls-certs\") pod \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\" (UID: \"9b097b84-fb8c-41f6-8b74-9f9a29245b59\") "
Nov 02 02:07:25.825283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825003    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19694632-5293-4f90-bd57-e118aa208699-kube-api-access-9m6w5\" (UniqueName: \"kubernetes.io/projected/19694632-5293-4f90-bd57-e118aa208699-kube-api-access-9m6w5\") pod \"19694632-5293-4f90-bd57-e118aa208699\" (UID: \"19694632-5293-4f90-bd57-e118aa208699\") "
Nov 02 02:07:25.825283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825151    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tmp-dir" pod "9b097b84-fb8c-41f6-8b74-9f9a29245b59" (UID: "9b097b84-fb8c-41f6-8b74-9f9a29245b59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.825283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825243    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-home" pod "19694632-5293-4f90-bd57-e118aa208699" (UID: "19694632-5293-4f90-bd57-e118aa208699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.825711 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825689    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.825801 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825716    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.825801 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825733    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.825801 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825747    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.825801 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825760    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.825801 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.825775    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.827298 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.827272    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-dshm" pod "9b097b84-fb8c-41f6-8b74-9f9a29245b59" (UID: "9b097b84-fb8c-41f6-8b74-9f9a29245b59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.827507 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.827483    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19694632-5293-4f90-bd57-e118aa208699-tls-certs" pod "19694632-5293-4f90-bd57-e118aa208699" (UID: "19694632-5293-4f90-bd57-e118aa208699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:25.827660 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.827596    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b097b84-fb8c-41f6-8b74-9f9a29245b59-kube-api-access-wxbvj" pod "9b097b84-fb8c-41f6-8b74-9f9a29245b59" (UID: "9b097b84-fb8c-41f6-8b74-9f9a29245b59"). InnerVolumeSpecName "kube-api-access-wxbvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:25.827879 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.827858    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tls-certs" pod "9b097b84-fb8c-41f6-8b74-9f9a29245b59" (UID: "9b097b84-fb8c-41f6-8b74-9f9a29245b59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:07:25.828105 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.827862    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19694632-5293-4f90-bd57-e118aa208699-kube-api-access-9m6w5" pod "19694632-5293-4f90-bd57-e118aa208699" (UID: "19694632-5293-4f90-bd57-e118aa208699"). InnerVolumeSpecName "kube-api-access-9m6w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:07:25.828105 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.828018    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-dshm" pod "19694632-5293-4f90-bd57-e118aa208699" (UID: "19694632-5293-4f90-bd57-e118aa208699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:07:25.926982 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.926920    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbvj\" (UniqueName: \"kubernetes.io/projected/9b097b84-fb8c-41f6-8b74-9f9a29245b59-kube-api-access-wxbvj\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.926982 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.926965    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b097b84-fb8c-41f6-8b74-9f9a29245b59-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.926982 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.926980    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19694632-5293-4f90-bd57-e118aa208699-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.926982 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.926992    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b097b84-fb8c-41f6-8b74-9f9a29245b59-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.927427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.927006    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6w5\" (UniqueName: \"kubernetes.io/projected/19694632-5293-4f90-bd57-e118aa208699-kube-api-access-9m6w5\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:25.927427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:25.927019    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19694632-5293-4f90-bd57-e118aa208699-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:07:26.316044 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.316013    4287 generic.go:356] "Generic (PLEG): container finished" podID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerID="ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5" exitCode=0
Nov 02 02:07:26.316221 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.316083    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" event={"ID":"0589391c-05a1-4ccf-b816-9615d5a6d69f","Type":"ContainerDied","Data":"ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5"}
Nov 02 02:07:26.316512 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.316471    4287 scope.go:122] "RemoveContainer" containerID="ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5"
Nov 02 02:07:26.317541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.317518    4287 generic.go:356] "Generic (PLEG): container finished" podID="19694632-5293-4f90-bd57-e118aa208699" containerID="9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098" exitCode=0
Nov 02 02:07:26.317656 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.317564    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"
Nov 02 02:07:26.317656 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.317576    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" event={"ID":"19694632-5293-4f90-bd57-e118aa208699","Type":"ContainerDied","Data":"9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098"}
Nov 02 02:07:26.317656 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.317631    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n" event={"ID":"19694632-5293-4f90-bd57-e118aa208699","Type":"ContainerDied","Data":"2e7c55beb45d721085f873f89cda3d4fca5838f50e35ef0afa5e436da0b0f8b4"}
Nov 02 02:07:26.317656 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.317648    4287 scope.go:122] "RemoveContainer" containerID="9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098"
Nov 02 02:07:26.318043 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.318011    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:07:26.318133 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.318105    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:07:26.318187 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.318149    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:07:26.320115 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.319370    4287 generic.go:356] "Generic (PLEG): container finished" podID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerID="b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709" exitCode=0
Nov 02 02:07:26.320115 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.319458    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" event={"ID":"9b097b84-fb8c-41f6-8b74-9f9a29245b59","Type":"ContainerDied","Data":"b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709"}
Nov 02 02:07:26.320115 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.319483    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42" event={"ID":"9b097b84-fb8c-41f6-8b74-9f9a29245b59","Type":"ContainerDied","Data":"da818661fcc9129e750b6fa1e36b69bfac51c8c5b158377460217ac455434462"}
Nov 02 02:07:26.320115 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.319540    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"
Nov 02 02:07:26.330079 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.330061    4287 scope.go:122] "RemoveContainer" containerID="9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098"
Nov 02 02:07:26.330406 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:07:26.330383    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098\": container with ID starting with 9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098 not found: ID does not exist" containerID="9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098"
Nov 02 02:07:26.330509 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.330417    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098"} err="failed to get container status \"9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098\": rpc error: code = NotFound desc = could not find container \"9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098\": container with ID starting with 9e9c6c6b912a44678298cc7b1e5579ced25947995f3663fd37b93af144edb098 not found: ID does not exist"
Nov 02 02:07:26.330509 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.330440    4287 scope.go:122] "RemoveContainer" containerID="b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709"
Nov 02 02:07:26.340246 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.340227    4287 scope.go:122] "RemoveContainer" containerID="b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709"
Nov 02 02:07:26.340501 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:07:26.340482    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709\": container with ID starting with b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709 not found: ID does not exist" containerID="b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709"
Nov 02 02:07:26.340586 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.340507    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709"} err="failed to get container status \"b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709\": rpc error: code = NotFound desc = could not find container \"b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709\": container with ID starting with b2373466e6feadec3f756d93157527ad83bf6d911157f690659b6553139bd709 not found: ID does not exist"
Nov 02 02:07:26.551037 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.550966    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"]
Nov 02 02:07:26.553661 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.553638    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-w2-36a1e0-kserve-84b85f9f77-jmg42"]
Nov 02 02:07:26.661631 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.661522    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"]
Nov 02 02:07:26.665746 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:26.665709    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/premium-w2-36a1e0-kserve-6d56bcf654-bzv9n"]
Nov 02 02:07:27.329695 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:27.329658    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" event={"ID":"0589391c-05a1-4ccf-b816-9615d5a6d69f","Type":"ContainerStarted","Data":"769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864"}
Nov 02 02:07:28.199688 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:28.199653    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:07:28.200084 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:28.199857    4287 patch_prober.go:29] interesting pod/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 02:07:28.200084 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:28.199914    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 02:07:28.242170 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:28.242140    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19694632-5293-4f90-bd57-e118aa208699" path="/var/lib/kubelet/pods/19694632-5293-4f90-bd57-e118aa208699/volumes"
Nov 02 02:07:28.242566 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:28.242552    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" path="/var/lib/kubelet/pods/9b097b84-fb8c-41f6-8b74-9f9a29245b59/volumes"
Nov 02 02:07:29.199303 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:29.199268    4287 patch_prober.go:29] interesting pod/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 02:07:29.199471 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:29.199319    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 02:07:30.199569 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:30.199534    4287 patch_prober.go:29] interesting pod/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 02:07:30.199994 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:30.199588    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 02:07:31.199500 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:31.199464    4287 patch_prober.go:29] interesting pod/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 02:07:31.199687 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:31.199524    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 02:07:32.199561 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:32.199521    4287 patch_prober.go:29] interesting pod/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 02 02:07:32.199752 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:32.199581    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 02 02:07:33.203815 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:33.203789    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:07:33.204213 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:33.204072    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:07:33.204658 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:07:33.204642    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:07:46.038620 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:07:46.038566    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:08:16.040099 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:08:16.040057    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:08:25.705055 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:25.705017    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"]
Nov 02 02:08:25.705510 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:25.705219    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d" podUID="12e4fa5c-9232-42d1-b6ea-cf9448c24602" containerName="maas-api" containerID="cri-o://2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e" gracePeriod=30
Nov 02 02:08:25.995864 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:25.995834    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 02:08:26.080305 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.080269    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e4fa5c-9232-42d1-b6ea-cf9448c24602-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e4fa5c-9232-42d1-b6ea-cf9448c24602-openshift-service-ca\") pod \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") "
Nov 02 02:08:26.080512 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.080380    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-metrics-certs\") pod \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") "
Nov 02 02:08:26.080587 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.080561    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls\") pod \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") "
Nov 02 02:08:26.080737 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.080716    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e4fa5c-9232-42d1-b6ea-cf9448c24602-kube-api-access-8qq2d\" (UniqueName: \"kubernetes.io/projected/12e4fa5c-9232-42d1-b6ea-cf9448c24602-kube-api-access-8qq2d\") pod \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\" (UID: \"12e4fa5c-9232-42d1-b6ea-cf9448c24602\") "
Nov 02 02:08:26.080992 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.080742    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e4fa5c-9232-42d1-b6ea-cf9448c24602-openshift-service-ca" pod "12e4fa5c-9232-42d1-b6ea-cf9448c24602" (UID: "12e4fa5c-9232-42d1-b6ea-cf9448c24602"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:26.081936 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.081820    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e4fa5c-9232-42d1-b6ea-cf9448c24602-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:26.094581 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.094539    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls" pod "12e4fa5c-9232-42d1-b6ea-cf9448c24602" (UID: "12e4fa5c-9232-42d1-b6ea-cf9448c24602"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:26.109881 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.101170    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-metrics-certs" pod "12e4fa5c-9232-42d1-b6ea-cf9448c24602" (UID: "12e4fa5c-9232-42d1-b6ea-cf9448c24602"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:26.110976 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.110935    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e4fa5c-9232-42d1-b6ea-cf9448c24602-kube-api-access-8qq2d" pod "12e4fa5c-9232-42d1-b6ea-cf9448c24602" (UID: "12e4fa5c-9232-42d1-b6ea-cf9448c24602"). InnerVolumeSpecName "kube-api-access-8qq2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:26.182724 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.182691    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:26.182724 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.182722    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qq2d\" (UniqueName: \"kubernetes.io/projected/12e4fa5c-9232-42d1-b6ea-cf9448c24602-kube-api-access-8qq2d\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:26.182927 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.182733    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e4fa5c-9232-42d1-b6ea-cf9448c24602-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:26.687886 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.687843    4287 generic.go:356] "Generic (PLEG): container finished" podID="12e4fa5c-9232-42d1-b6ea-cf9448c24602" containerID="2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e" exitCode=0
Nov 02 02:08:26.688094 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.687921    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d" event={"ID":"12e4fa5c-9232-42d1-b6ea-cf9448c24602","Type":"ContainerDied","Data":"2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e"}
Nov 02 02:08:26.688094 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.687977    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d" event={"ID":"12e4fa5c-9232-42d1-b6ea-cf9448c24602","Type":"ContainerDied","Data":"f1c31389603e8a4e0d08dfef3c531b03eff742361a5ce9e796d98a68f5082356"}
Nov 02 02:08:26.688094 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.688004    4287 scope.go:122] "RemoveContainer" containerID="2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e"
Nov 02 02:08:26.691148 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.691124    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"
Nov 02 02:08:26.703239 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.703216    4287 scope.go:122] "RemoveContainer" containerID="2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e"
Nov 02 02:08:26.703587 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:08:26.703556    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e\": container with ID starting with 2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e not found: ID does not exist" containerID="2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e"
Nov 02 02:08:26.703689 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.703593    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e"} err="failed to get container status \"2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e\": rpc error: code = NotFound desc = could not find container \"2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e\": container with ID starting with 2cc1ae0da151af3160226e886e7308367390d208fcd90f1e65b55157a0df547e not found: ID does not exist"
Nov 02 02:08:26.860912 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.860866    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"]
Nov 02 02:08:26.865774 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:26.865747    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-36a1e0-58fb644f5c-vp59d"]
Nov 02 02:08:28.237928 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:28.237894    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e4fa5c-9232-42d1-b6ea-cf9448c24602" path="/var/lib/kubelet/pods/12e4fa5c-9232-42d1-b6ea-cf9448c24602/volumes"
Nov 02 02:08:30.313171 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.313132    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"]
Nov 02 02:08:30.313812 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.313791    4287 cpu_manager.go:414] "containerMap: removing container" podUID="12e4fa5c-9232-42d1-b6ea-cf9448c24602" containerName="maas-api"
Nov 02 02:08:30.313922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.313813    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e4fa5c-9232-42d1-b6ea-cf9448c24602" containerName="maas-api"
Nov 02 02:08:30.313922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.313843    4287 cpu_manager.go:414] "containerMap: removing container" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main"
Nov 02 02:08:30.313922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.313852    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main"
Nov 02 02:08:30.313922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.313884    4287 cpu_manager.go:414] "containerMap: removing container" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main"
Nov 02 02:08:30.313922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.313892    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main"
Nov 02 02:08:30.314176 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.313999    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b097b84-fb8c-41f6-8b74-9f9a29245b59" containerName="main"
Nov 02 02:08:30.314176 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.314014    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="19694632-5293-4f90-bd57-e118aa208699" containerName="main"
Nov 02 02:08:30.314176 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.314028    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e4fa5c-9232-42d1-b6ea-cf9448c24602" containerName="maas-api"
Nov 02 02:08:30.318685 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.318660    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.418845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.418814    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d0fdb9\"/\"openshift-service-ca.crt\""
Nov 02 02:08:30.421376 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.421355    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d0fdb9\"/\"default-dockercfg-vx79x\""
Nov 02 02:08:30.421489 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.421443    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d0fdb9\"/\"kube-root-ca.crt\""
Nov 02 02:08:30.421704 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.421687    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d0fdb9\"/\"test-model-d0fdb9-kserve-self-signed-certs\""
Nov 02 02:08:30.424545 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.424523    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"]
Nov 02 02:08:30.429515 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.429494    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmmn\" (UniqueName: \"kubernetes.io/projected/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-kube-api-access-vlmmn\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.429590 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.429537    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-home\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.429590 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.429555    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-model-cache\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.429792 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.429773    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tmp-dir\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.429856 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.429810    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tls-certs\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.429856 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.429847    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-dshm\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.531364 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.531329    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tmp-dir\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.531364 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.531368    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tls-certs\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.531652 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.531388    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-dshm\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.531652 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.531497    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmmn\" (UniqueName: \"kubernetes.io/projected/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-kube-api-access-vlmmn\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.531652 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.531540    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-home\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.531652 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.531564    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-model-cache\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.531873 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.531852    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tmp-dir\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.531978 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.531954    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-model-cache\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.532094 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.532062    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-home\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.534546 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.534522    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-dshm\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.534812 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.534795    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tls-certs\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.592968 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.592894    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmmn\" (UniqueName: \"kubernetes.io/projected/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-kube-api-access-vlmmn\") pod \"test-model-d0fdb9-kserve-7bc96cd98f-tnlr4\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") " pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:30.631847 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:30.631788    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:31.057783 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:31.057751    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"]
Nov 02 02:08:31.059278 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:08:31.059251    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf5bc1f_48d5_4811_bd1a_c3939e31cb29.slice/crio-149593db865e24802524de1cf1eca91d3b67249b62568a6ba23577e8ec17824c WatchSource:0}: Error finding container 149593db865e24802524de1cf1eca91d3b67249b62568a6ba23577e8ec17824c: Status 404 returned error can't find the container with id 149593db865e24802524de1cf1eca91d3b67249b62568a6ba23577e8ec17824c
Nov 02 02:08:31.739180 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:31.739144    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4" event={"ID":"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29","Type":"ContainerStarted","Data":"9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db"}
Nov 02 02:08:31.739180 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:31.739187    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4" event={"ID":"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29","Type":"ContainerStarted","Data":"149593db865e24802524de1cf1eca91d3b67249b62568a6ba23577e8ec17824c"}
Nov 02 02:08:31.857450 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:31.857396    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4" podStartSLOduration=1.857379398 podStartE2EDuration="1.857379398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:31.855882696 +0000 UTC m=+2619.729523748" watchObservedRunningTime="2026-11-02 02:08:31.857379398 +0000 UTC m=+2619.731020451"
Nov 02 02:08:31.980085 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:31.979877    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"]
Nov 02 02:08:31.982292 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:31.980229    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9" podUID="e60c46d9-232c-4ba5-a854-19eaea3c508d" containerName="maas-api" containerID="cri-o://403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140" gracePeriod=30
Nov 02 02:08:32.287710 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.287676    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:08:32.353270 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.351652    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls\") pod \"e60c46d9-232c-4ba5-a854-19eaea3c508d\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") "
Nov 02 02:08:32.353270 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.351758    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-metrics-certs\") pod \"e60c46d9-232c-4ba5-a854-19eaea3c508d\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") "
Nov 02 02:08:32.353270 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.351858    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60c46d9-232c-4ba5-a854-19eaea3c508d-kube-api-access-b926b\" (UniqueName: \"kubernetes.io/projected/e60c46d9-232c-4ba5-a854-19eaea3c508d-kube-api-access-b926b\") pod \"e60c46d9-232c-4ba5-a854-19eaea3c508d\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") "
Nov 02 02:08:32.353270 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.351946    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60c46d9-232c-4ba5-a854-19eaea3c508d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60c46d9-232c-4ba5-a854-19eaea3c508d-openshift-service-ca\") pod \"e60c46d9-232c-4ba5-a854-19eaea3c508d\" (UID: \"e60c46d9-232c-4ba5-a854-19eaea3c508d\") "
Nov 02 02:08:32.353646 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.353324    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60c46d9-232c-4ba5-a854-19eaea3c508d-openshift-service-ca" pod "e60c46d9-232c-4ba5-a854-19eaea3c508d" (UID: "e60c46d9-232c-4ba5-a854-19eaea3c508d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:32.356063 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.356029    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60c46d9-232c-4ba5-a854-19eaea3c508d-kube-api-access-b926b" pod "e60c46d9-232c-4ba5-a854-19eaea3c508d" (UID: "e60c46d9-232c-4ba5-a854-19eaea3c508d"). InnerVolumeSpecName "kube-api-access-b926b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:32.356183 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.356109    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-metrics-certs" pod "e60c46d9-232c-4ba5-a854-19eaea3c508d" (UID: "e60c46d9-232c-4ba5-a854-19eaea3c508d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:32.367632 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.357960    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls" pod "e60c46d9-232c-4ba5-a854-19eaea3c508d" (UID: "e60c46d9-232c-4ba5-a854-19eaea3c508d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:08:32.453405 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.453371    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:32.453405 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.453397    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e60c46d9-232c-4ba5-a854-19eaea3c508d-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:32.453405 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.453409    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b926b\" (UniqueName: \"kubernetes.io/projected/e60c46d9-232c-4ba5-a854-19eaea3c508d-kube-api-access-b926b\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:32.453758 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.453420    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e60c46d9-232c-4ba5-a854-19eaea3c508d-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:32.631882 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.631840    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:32.745772 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.745740    4287 generic.go:356] "Generic (PLEG): container finished" podID="e60c46d9-232c-4ba5-a854-19eaea3c508d" containerID="403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140" exitCode=0
Nov 02 02:08:32.746165 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.745794    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"
Nov 02 02:08:32.746165 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.745869    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9" event={"ID":"e60c46d9-232c-4ba5-a854-19eaea3c508d","Type":"ContainerDied","Data":"403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140"}
Nov 02 02:08:32.746165 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.745917    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9" event={"ID":"e60c46d9-232c-4ba5-a854-19eaea3c508d","Type":"ContainerDied","Data":"daa03de47e8a8458a97954f1d7d12d7aa42cb70b3806dc6ceeee5dc6cefe8d5a"}
Nov 02 02:08:32.746165 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.745946    4287 scope.go:122] "RemoveContainer" containerID="403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140"
Nov 02 02:08:32.756501 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.756485    4287 scope.go:122] "RemoveContainer" containerID="403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140"
Nov 02 02:08:32.756800 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:08:32.756782    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140\": container with ID starting with 403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140 not found: ID does not exist" containerID="403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140"
Nov 02 02:08:32.756857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.756811    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140"} err="failed to get container status \"403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140\": rpc error: code = NotFound desc = could not find container \"403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140\": container with ID starting with 403ac2103e6f9dd15dff7e6efea532a752437f8274bfbf0cb5209d1d43736140 not found: ID does not exist"
Nov 02 02:08:32.927330 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.927232    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"]
Nov 02 02:08:32.929643 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:32.928026    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c954d-5d57f645bb-x8wt9"]
Nov 02 02:08:34.237791 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.237756    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60c46d9-232c-4ba5-a854-19eaea3c508d" path="/var/lib/kubelet/pods/e60c46d9-232c-4ba5-a854-19eaea3c508d/volumes"
Nov 02 02:08:34.802126 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.802092    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"]
Nov 02 02:08:34.802562 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.802550    4287 cpu_manager.go:414] "containerMap: removing container" podUID="e60c46d9-232c-4ba5-a854-19eaea3c508d" containerName="maas-api"
Nov 02 02:08:34.802621 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.802564    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60c46d9-232c-4ba5-a854-19eaea3c508d" containerName="maas-api"
Nov 02 02:08:34.802685 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.802675    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60c46d9-232c-4ba5-a854-19eaea3c508d" containerName="maas-api"
Nov 02 02:08:34.807154 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.807137    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.878656 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.878595    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-home\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.878822 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.878719    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdgl\" (UniqueName: \"kubernetes.io/projected/be0c997b-eb96-467b-a3e2-84fd7450db9a-kube-api-access-xwdgl\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.878865 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.878814    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-tmp-dir\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.878865 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.878838    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0c997b-eb96-467b-a3e2-84fd7450db9a-tls-certs\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.878932 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.878870    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-dshm\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.878975 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.878961    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-model-cache\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.911484 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.911429    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-361fcc\"/\"test-model-361fcc-kserve-self-signed-certs\""
Nov 02 02:08:34.911704 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.911511    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-361fcc\"/\"kube-root-ca.crt\""
Nov 02 02:08:34.911704 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.911528    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-361fcc\"/\"default-dockercfg-hfqrq\""
Nov 02 02:08:34.911704 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.911536    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-361fcc\"/\"openshift-service-ca.crt\""
Nov 02 02:08:34.914508 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.914483    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"]
Nov 02 02:08:34.980295 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980264    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-tmp-dir\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.980295 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980297    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0c997b-eb96-467b-a3e2-84fd7450db9a-tls-certs\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.980561 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980329    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-dshm\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.980561 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980384    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-model-cache\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.980561 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980415    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-home\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.980561 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980445    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdgl\" (UniqueName: \"kubernetes.io/projected/be0c997b-eb96-467b-a3e2-84fd7450db9a-kube-api-access-xwdgl\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.980849 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980827    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-model-cache\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.980914 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980874    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-tmp-dir\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.980914 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.980889    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-home\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.983078 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.983052    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-dshm\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:34.984689 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:34.984669    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0c997b-eb96-467b-a3e2-84fd7450db9a-tls-certs\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:35.042379 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:35.042341    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdgl\" (UniqueName: \"kubernetes.io/projected/be0c997b-eb96-467b-a3e2-84fd7450db9a-kube-api-access-xwdgl\") pod \"test-model-361fcc-kserve-7fbdfd6d55-z6qb6\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") " pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:35.119382 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:35.119183    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:35.133658 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:35.124903    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"]
Nov 02 02:08:35.133658 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:35.125468    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy" containerID="cri-o://c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f" gracePeriod=30
Nov 02 02:08:35.522179 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:35.522151    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"]
Nov 02 02:08:35.522820 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:08:35.522795    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe0c997b_eb96_467b_a3e2_84fd7450db9a.slice/crio-b17c10dc352955c49b21dc4fc551a914f8d988b5176fe7d3eae571e65922e278 WatchSource:0}: Error finding container b17c10dc352955c49b21dc4fc551a914f8d988b5176fe7d3eae571e65922e278: Status 404 returned error can't find the container with id b17c10dc352955c49b21dc4fc551a914f8d988b5176fe7d3eae571e65922e278
Nov 02 02:08:35.768068 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:35.768036    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6" event={"ID":"be0c997b-eb96-467b-a3e2-84fd7450db9a","Type":"ContainerStarted","Data":"f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0"}
Nov 02 02:08:35.768255 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:35.768077    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6" event={"ID":"be0c997b-eb96-467b-a3e2-84fd7450db9a","Type":"ContainerStarted","Data":"b17c10dc352955c49b21dc4fc551a914f8d988b5176fe7d3eae571e65922e278"}
Nov 02 02:08:35.892128 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:35.892014    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6" podStartSLOduration=1.891997249 podStartE2EDuration="1.891997249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:08:35.89029678 +0000 UTC m=+2623.763937880" watchObservedRunningTime="2026-11-02 02:08:35.891997249 +0000 UTC m=+2623.765638300"
Nov 02 02:08:36.119919 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:36.119884    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:40.382597 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.382570    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 02:08:40.456051 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.455987    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-certs\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456051 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456045    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-token\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-token\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456260 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456096    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-socket\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456260 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456145    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-kube-api-access-wx55q\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-kube-api-access-wx55q\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456260 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456176    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-envoy\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456260 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456233    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-data\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456472 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456276    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-certs" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:40.456472 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456309    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-credential-socket\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456472 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456340    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-podinfo\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456472 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456402    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3b16cf-2480-4289-ab1a-e429adb66645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3b16cf-2480-4289-ab1a-e429adb66645-istiod-ca-cert\") pod \"bd3b16cf-2480-4289-ab1a-e429adb66645\" (UID: \"bd3b16cf-2480-4289-ab1a-e429adb66645\") "
Nov 02 02:08:40.456723 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456511    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-socket" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:40.456723 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456540    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-data" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:40.456723 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456586    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-credential-socket" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:40.456888 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456860    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3b16cf-2480-4289-ab1a-e429adb66645-istiod-ca-cert" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:08:40.457064 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.456998    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.457064 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.457024    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.457064 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.457038    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3b16cf-2480-4289-ab1a-e429adb66645-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.457064 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.457052    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.457064 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.457066    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.458499 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.458474    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-token" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:40.458592 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.458569    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-kube-api-access-wx55q" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "kube-api-access-wx55q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:08:40.458679 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.458640    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-envoy" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:08:40.458723 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.458697    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-podinfo" pod "bd3b16cf-2480-4289-ab1a-e429adb66645" (UID: "bd3b16cf-2480-4289-ab1a-e429adb66645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:08:40.558158 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.558123    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx55q\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-kube-api-access-wx55q\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.558158 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.558158    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.558329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.558173    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.558329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.558188    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3b16cf-2480-4289-ab1a-e429adb66645-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:08:40.632174 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.632140    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:40.644524 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.644501    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:40.796048 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.796018    4287 generic.go:356] "Generic (PLEG): container finished" podID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerID="c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f" exitCode=0
Nov 02 02:08:40.796229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.796066    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"
Nov 02 02:08:40.796229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.796111    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" event={"ID":"bd3b16cf-2480-4289-ab1a-e429adb66645","Type":"ContainerDied","Data":"c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f"}
Nov 02 02:08:40.796229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.796161    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz" event={"ID":"bd3b16cf-2480-4289-ab1a-e429adb66645","Type":"ContainerDied","Data":"bb667df5b68dfaabf9730ea4679af1480670ee1ad0a1b6d854144c410d527fd4"}
Nov 02 02:08:40.796229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.796190    4287 scope.go:122] "RemoveContainer" containerID="c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f"
Nov 02 02:08:40.808405 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.808389    4287 scope.go:122] "RemoveContainer" containerID="c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f"
Nov 02 02:08:40.808686 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:08:40.808665    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f\": container with ID starting with c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f not found: ID does not exist" containerID="c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f"
Nov 02 02:08:40.808739 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.808693    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f"} err="failed to get container status \"c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f\": rpc error: code = NotFound desc = could not find container \"c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f\": container with ID starting with c3ea01bb2277cf9e49756ef2df7436b63e79aa3899cc6e7d2a88d2155224078f not found: ID does not exist"
Nov 02 02:08:40.810285 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:40.810261    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:08:41.076892 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:41.076814    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"]
Nov 02 02:08:41.081447 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:41.081422    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-rcwvz"]
Nov 02 02:08:42.238032 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:42.237999    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" path="/var/lib/kubelet/pods/bd3b16cf-2480-4289-ab1a-e429adb66645/volumes"
Nov 02 02:08:45.119890 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:45.119848    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:45.133082 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:45.133044    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:45.848325 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:08:45.848294    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:08:46.038558 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:08:46.038513    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:09:16.039132 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:09:16.039091    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:09:46.038939 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:09:46.038896    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:09:52.619046 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:09:52.619012    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:09:52.636750 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:09:52.636727    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:10:16.038198 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:10:16.038154    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:10:22.250662 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.250629    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"]
Nov 02 02:10:22.251213 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.250833    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft" podUID="d32a2a55-b4ef-4548-87e5-03df6bc720e6" containerName="maas-api" containerID="cri-o://5228f3c0ff56d46ea66dbe5f36fd55839a6216f85428286e1d4b97e66fb68864" gracePeriod=30
Nov 02 02:10:22.378192 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.378161    4287 generic.go:356] "Generic (PLEG): container finished" podID="d32a2a55-b4ef-4548-87e5-03df6bc720e6" containerID="5228f3c0ff56d46ea66dbe5f36fd55839a6216f85428286e1d4b97e66fb68864" exitCode=0
Nov 02 02:10:22.378299 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.378213    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft" event={"ID":"d32a2a55-b4ef-4548-87e5-03df6bc720e6","Type":"ContainerDied","Data":"5228f3c0ff56d46ea66dbe5f36fd55839a6216f85428286e1d4b97e66fb68864"}
Nov 02 02:10:22.524280 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.524255    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:10:22.610505 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.610464    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32a2a55-b4ef-4548-87e5-03df6bc720e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2a55-b4ef-4548-87e5-03df6bc720e6-openshift-service-ca\") pod \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") "
Nov 02 02:10:22.610829 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.610806    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-metrics-certs\") pod \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") "
Nov 02 02:10:22.610981 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.610925    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32a2a55-b4ef-4548-87e5-03df6bc720e6-openshift-service-ca" pod "d32a2a55-b4ef-4548-87e5-03df6bc720e6" (UID: "d32a2a55-b4ef-4548-87e5-03df6bc720e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:22.610981 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.610954    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-maas-api-tls\") pod \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") "
Nov 02 02:10:22.611184 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.611116    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32a2a55-b4ef-4548-87e5-03df6bc720e6-kube-api-access-z5tvn\" (UniqueName: \"kubernetes.io/projected/d32a2a55-b4ef-4548-87e5-03df6bc720e6-kube-api-access-z5tvn\") pod \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\" (UID: \"d32a2a55-b4ef-4548-87e5-03df6bc720e6\") "
Nov 02 02:10:22.611527 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.611488    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d32a2a55-b4ef-4548-87e5-03df6bc720e6-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:22.613495 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.613469    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-metrics-certs" pod "d32a2a55-b4ef-4548-87e5-03df6bc720e6" (UID: "d32a2a55-b4ef-4548-87e5-03df6bc720e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:22.613649 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.613596    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-maas-api-tls" pod "d32a2a55-b4ef-4548-87e5-03df6bc720e6" (UID: "d32a2a55-b4ef-4548-87e5-03df6bc720e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:22.613766 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.613741    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32a2a55-b4ef-4548-87e5-03df6bc720e6-kube-api-access-z5tvn" pod "d32a2a55-b4ef-4548-87e5-03df6bc720e6" (UID: "d32a2a55-b4ef-4548-87e5-03df6bc720e6"). InnerVolumeSpecName "kube-api-access-z5tvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:22.712619 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.712584    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5tvn\" (UniqueName: \"kubernetes.io/projected/d32a2a55-b4ef-4548-87e5-03df6bc720e6-kube-api-access-z5tvn\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:22.712785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.712630    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:22.712785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:22.712641    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d32a2a55-b4ef-4548-87e5-03df6bc720e6-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:23.385233 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:23.385196    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"
Nov 02 02:10:23.385233 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:23.385206    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft" event={"ID":"d32a2a55-b4ef-4548-87e5-03df6bc720e6","Type":"ContainerDied","Data":"118b477a3bd7b2b276df17db1b8535725dd864ddfc91d5e3b1392b4de77d5eb0"}
Nov 02 02:10:23.385799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:23.385257    4287 scope.go:122] "RemoveContainer" containerID="5228f3c0ff56d46ea66dbe5f36fd55839a6216f85428286e1d4b97e66fb68864"
Nov 02 02:10:23.550028 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:23.549992    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"]
Nov 02 02:10:23.599466 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:23.599428    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c78625-6459476c9-qj9ft"]
Nov 02 02:10:24.238887 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:24.238854    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32a2a55-b4ef-4548-87e5-03df6bc720e6" path="/var/lib/kubelet/pods/d32a2a55-b4ef-4548-87e5-03df6bc720e6/volumes"
Nov 02 02:10:36.492285 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:36.492196    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"]
Nov 02 02:10:46.038162 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:10:46.038120    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:10:51.512757 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.512712    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main" containerID="cri-o://711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff" gracePeriod=45
Nov 02 02:10:51.767052 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.766988    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 02:10:51.831309 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831278    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tmp-dir\") pod \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") "
Nov 02 02:10:51.831498 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831330    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tls-certs\") pod \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") "
Nov 02 02:10:51.831498 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831361    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-dshm\") pod \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") "
Nov 02 02:10:51.831498 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831398    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-model-cache\") pod \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") "
Nov 02 02:10:51.831498 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831417    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-home\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-home\") pod \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") "
Nov 02 02:10:51.831498 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831459    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351cb013-4b1d-4d42-b1d8-9e45632c86d5-kube-api-access-mzvh2\" (UniqueName: \"kubernetes.io/projected/351cb013-4b1d-4d42-b1d8-9e45632c86d5-kube-api-access-mzvh2\") pod \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\" (UID: \"351cb013-4b1d-4d42-b1d8-9e45632c86d5\") "
Nov 02 02:10:51.831815 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831570    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tmp-dir" pod "351cb013-4b1d-4d42-b1d8-9e45632c86d5" (UID: "351cb013-4b1d-4d42-b1d8-9e45632c86d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.831815 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831734    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-model-cache" pod "351cb013-4b1d-4d42-b1d8-9e45632c86d5" (UID: "351cb013-4b1d-4d42-b1d8-9e45632c86d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.831815 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831758    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-home" pod "351cb013-4b1d-4d42-b1d8-9e45632c86d5" (UID: "351cb013-4b1d-4d42-b1d8-9e45632c86d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.832012 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.831991    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.832056 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.832020    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.832056 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.832037    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.833988 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.833960    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351cb013-4b1d-4d42-b1d8-9e45632c86d5-kube-api-access-mzvh2" pod "351cb013-4b1d-4d42-b1d8-9e45632c86d5" (UID: "351cb013-4b1d-4d42-b1d8-9e45632c86d5"). InnerVolumeSpecName "kube-api-access-mzvh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:51.834229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.834207    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tls-certs" pod "351cb013-4b1d-4d42-b1d8-9e45632c86d5" (UID: "351cb013-4b1d-4d42-b1d8-9e45632c86d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:51.834339 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.834222    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-dshm" pod "351cb013-4b1d-4d42-b1d8-9e45632c86d5" (UID: "351cb013-4b1d-4d42-b1d8-9e45632c86d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:10:51.932809 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.932778    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351cb013-4b1d-4d42-b1d8-9e45632c86d5-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.932809 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.932807    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351cb013-4b1d-4d42-b1d8-9e45632c86d5-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:51.933008 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:51.932820    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzvh2\" (UniqueName: \"kubernetes.io/projected/351cb013-4b1d-4d42-b1d8-9e45632c86d5-kube-api-access-mzvh2\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:52.547756 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.547723    4287 generic.go:356] "Generic (PLEG): container finished" podID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerID="711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff" exitCode=0
Nov 02 02:10:52.548207 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.547770    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" event={"ID":"351cb013-4b1d-4d42-b1d8-9e45632c86d5","Type":"ContainerDied","Data":"711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff"}
Nov 02 02:10:52.548207 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.547788    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"
Nov 02 02:10:52.548207 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.547808    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt" event={"ID":"351cb013-4b1d-4d42-b1d8-9e45632c86d5","Type":"ContainerDied","Data":"b3e30f2a2abe9c68a1279fd6a90f66d17445b1c6b582970ec6f7c1de8d3f5606"}
Nov 02 02:10:52.548207 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.547827    4287 scope.go:122] "RemoveContainer" containerID="711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff"
Nov 02 02:10:52.557307 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.557286    4287 scope.go:122] "RemoveContainer" containerID="711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff"
Nov 02 02:10:52.557602 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:10:52.557583    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff\": container with ID starting with 711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff not found: ID does not exist" containerID="711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff"
Nov 02 02:10:52.557684 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.557631    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff"} err="failed to get container status \"711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff\": rpc error: code = NotFound desc = could not find container \"711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff\": container with ID starting with 711fefc413c493349a0f3319b02a136b91ad32b29102231cc7ff9b0375952aff not found: ID does not exist"
Nov 02 02:10:52.717793 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.717760    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"]
Nov 02 02:10:52.767076 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:52.767042    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-w6-5888dc-kserve-67bc57f8d7-c7vkt"]
Nov 02 02:10:54.238780 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:54.238745    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" path="/var/lib/kubelet/pods/351cb013-4b1d-4d42-b1d8-9e45632c86d5/volumes"
Nov 02 02:10:58.255489 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.255450    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"]
Nov 02 02:10:58.256015 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.255739    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6" podUID="4b496963-737d-4d4b-b618-ad31f1305896" containerName="maas-api" containerID="cri-o://0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f" gracePeriod=30
Nov 02 02:10:58.535395 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.535361    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:10:58.583388 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.583350    4287 generic.go:356] "Generic (PLEG): container finished" podID="4b496963-737d-4d4b-b618-ad31f1305896" containerID="0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f" exitCode=0
Nov 02 02:10:58.583583 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.583395    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"
Nov 02 02:10:58.583583 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.583432    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6" event={"ID":"4b496963-737d-4d4b-b618-ad31f1305896","Type":"ContainerDied","Data":"0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f"}
Nov 02 02:10:58.583583 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.583507    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6" event={"ID":"4b496963-737d-4d4b-b618-ad31f1305896","Type":"ContainerDied","Data":"f20dfb50aeac6bbe92c52af5e377893f394dcd6e1608d6718cb385b9891d70fb"}
Nov 02 02:10:58.583583 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.583519    4287 scope.go:122] "RemoveContainer" containerID="0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f"
Nov 02 02:10:58.593535 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.593518    4287 scope.go:122] "RemoveContainer" containerID="0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f"
Nov 02 02:10:58.593838 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:10:58.593817    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f\": container with ID starting with 0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f not found: ID does not exist" containerID="0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f"
Nov 02 02:10:58.593924 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.593849    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f"} err="failed to get container status \"0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f\": rpc error: code = NotFound desc = could not find container \"0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f\": container with ID starting with 0c51111c0ebfcfd5075c3b85a3797acb82be1f55ae4c10e0bf5ded2cb455941f not found: ID does not exist"
Nov 02 02:10:58.597157 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.597138    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b496963-737d-4d4b-b618-ad31f1305896-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b496963-737d-4d4b-b618-ad31f1305896-openshift-service-ca\") pod \"4b496963-737d-4d4b-b618-ad31f1305896\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") "
Nov 02 02:10:58.597240 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.597230    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b496963-737d-4d4b-b618-ad31f1305896-kube-api-access-qkmk7\" (UniqueName: \"kubernetes.io/projected/4b496963-737d-4d4b-b618-ad31f1305896-kube-api-access-qkmk7\") pod \"4b496963-737d-4d4b-b618-ad31f1305896\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") "
Nov 02 02:10:58.597357 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.597333    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-metrics-certs\") pod \"4b496963-737d-4d4b-b618-ad31f1305896\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") "
Nov 02 02:10:58.597409 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.597385    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls\") pod \"4b496963-737d-4d4b-b618-ad31f1305896\" (UID: \"4b496963-737d-4d4b-b618-ad31f1305896\") "
Nov 02 02:10:58.597562 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.597539    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b496963-737d-4d4b-b618-ad31f1305896-openshift-service-ca" pod "4b496963-737d-4d4b-b618-ad31f1305896" (UID: "4b496963-737d-4d4b-b618-ad31f1305896"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:10:58.597914 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.597891    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b496963-737d-4d4b-b618-ad31f1305896-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:58.599557 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.599529    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b496963-737d-4d4b-b618-ad31f1305896-kube-api-access-qkmk7" pod "4b496963-737d-4d4b-b618-ad31f1305896" (UID: "4b496963-737d-4d4b-b618-ad31f1305896"). InnerVolumeSpecName "kube-api-access-qkmk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:10:58.599682 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.599646    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls" pod "4b496963-737d-4d4b-b618-ad31f1305896" (UID: "4b496963-737d-4d4b-b618-ad31f1305896"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:58.600035 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.600001    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-metrics-certs" pod "4b496963-737d-4d4b-b618-ad31f1305896" (UID: "4b496963-737d-4d4b-b618-ad31f1305896"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:10:58.699283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.699250    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkmk7\" (UniqueName: \"kubernetes.io/projected/4b496963-737d-4d4b-b618-ad31f1305896-kube-api-access-qkmk7\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:58.699283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.699279    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:58.699283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:58.699289    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b496963-737d-4d4b-b618-ad31f1305896-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:10:59.004320 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:59.004287    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"]
Nov 02 02:10:59.006960 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:10:59.006927    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9039bf-56d8557bff-mndk6"]
Nov 02 02:11:00.237855 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:00.237825    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b496963-737d-4d4b-b618-ad31f1305896" path="/var/lib/kubelet/pods/4b496963-737d-4d4b-b618-ad31f1305896/volumes"
Nov 02 02:11:07.270595 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:07.270554    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"]
Nov 02 02:11:07.271011 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:07.270840    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy" containerID="cri-o://769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864" gracePeriod=30
Nov 02 02:11:12.529781 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.529753    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:11:12.642036 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.641953    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-kube-api-access-shn68\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-kube-api-access-shn68\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642036 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642016    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-certs\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642270 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642119    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-credential-socket\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642270 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642181    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-token\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-token\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642270 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642211    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-socket\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642298    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-certs" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:12.642439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642327    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-podinfo\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642366    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0589391c-05a1-4ccf-b816-9615d5a6d69f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589391c-05a1-4ccf-b816-9615d5a6d69f-istiod-ca-cert\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642386    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-credential-socket" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:12.642439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642418    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-data\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642725 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642472    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-envoy\") pod \"0589391c-05a1-4ccf-b816-9615d5a6d69f\" (UID: \"0589391c-05a1-4ccf-b816-9615d5a6d69f\") "
Nov 02 02:11:12.642725 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642542    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-socket" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:12.642725 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642679    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-data" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:12.643057 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642870    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0589391c-05a1-4ccf-b816-9615d5a6d69f-istiod-ca-cert" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:12.643057 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642918    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.643057 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642937    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.643057 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642956    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.643057 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.642976    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.644493 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.644461    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-podinfo" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:12.644622 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.644551    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-kube-api-access-shn68" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "kube-api-access-shn68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:12.644622 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.644547    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-token" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:12.644712 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.644668    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-envoy" pod "0589391c-05a1-4ccf-b816-9615d5a6d69f" (UID: "0589391c-05a1-4ccf-b816-9615d5a6d69f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:12.663630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.663578    4287 generic.go:356] "Generic (PLEG): container finished" podID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerID="769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864" exitCode=0
Nov 02 02:11:12.663740 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.663651    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" event={"ID":"0589391c-05a1-4ccf-b816-9615d5a6d69f","Type":"ContainerDied","Data":"769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864"}
Nov 02 02:11:12.663740 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.663702    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9" event={"ID":"0589391c-05a1-4ccf-b816-9615d5a6d69f","Type":"ContainerDied","Data":"ab90e139000b05e81f45affa24128c7ae391db182ff52f914a9e03118f8cd5d0"}
Nov 02 02:11:12.663740 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.663658    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"
Nov 02 02:11:12.663893 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.663724    4287 scope.go:122] "RemoveContainer" containerID="769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864"
Nov 02 02:11:12.674890 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.674871    4287 scope.go:122] "RemoveContainer" containerID="ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5"
Nov 02 02:11:12.684299 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.684281    4287 scope.go:122] "RemoveContainer" containerID="769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864"
Nov 02 02:11:12.684577 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:11:12.684554    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864\": container with ID starting with 769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864 not found: ID does not exist" containerID="769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864"
Nov 02 02:11:12.684685 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.684583    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864"} err="failed to get container status \"769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864\": rpc error: code = NotFound desc = could not find container \"769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864\": container with ID starting with 769a31264977a8a09da5dd0b01c9716c70261f84518d930999e3471f0eedd864 not found: ID does not exist"
Nov 02 02:11:12.684685 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.684601    4287 scope.go:122] "RemoveContainer" containerID="ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5"
Nov 02 02:11:12.684860 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:11:12.684843    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5\": container with ID starting with ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5 not found: ID does not exist" containerID="ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5"
Nov 02 02:11:12.684911 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.684866    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5"} err="failed to get container status \"ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5\": rpc error: code = NotFound desc = could not find container \"ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5\": container with ID starting with ae97584271c75db4dde677c19c5e81da376d34cdc1bc9dfc9d5e5083036b38c5 not found: ID does not exist"
Nov 02 02:11:12.743922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.743880    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.743922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.743909    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0589391c-05a1-4ccf-b816-9615d5a6d69f-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.743922 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.743927    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.744145 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.743936    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shn68\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-kube-api-access-shn68\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.744145 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.743946    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0589391c-05a1-4ccf-b816-9615d5a6d69f-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:12.753924 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.753900    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"]
Nov 02 02:11:12.813206 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.813174    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"]
Nov 02 02:11:12.821733 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:12.821706    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9039bf-openshift-default-67564774b9-mn2w9"]
Nov 02 02:11:14.237511 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:14.237479    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" path="/var/lib/kubelet/pods/0589391c-05a1-4ccf-b816-9615d5a6d69f/volumes"
Nov 02 02:11:16.040090 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:11:16.040055    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:11:27.775920 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:27.775875    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main" containerID="cri-o://e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a" gracePeriod=45
Nov 02 02:11:28.036969 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.036915    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 02:11:28.111889 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.111858    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-model-cache\") pod \"05a9650a-d6e0-4509-a65b-4b60e98107da\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") "
Nov 02 02:11:28.112065 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.111919    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a9650a-d6e0-4509-a65b-4b60e98107da-tls-certs\" (UniqueName: \"kubernetes.io/secret/05a9650a-d6e0-4509-a65b-4b60e98107da-tls-certs\") pod \"05a9650a-d6e0-4509-a65b-4b60e98107da\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") "
Nov 02 02:11:28.112065 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.112009    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-home\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-home\") pod \"05a9650a-d6e0-4509-a65b-4b60e98107da\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") "
Nov 02 02:11:28.112065 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.112050    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-tmp-dir\") pod \"05a9650a-d6e0-4509-a65b-4b60e98107da\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") "
Nov 02 02:11:28.112227 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.112180    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-dshm\") pod \"05a9650a-d6e0-4509-a65b-4b60e98107da\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") "
Nov 02 02:11:28.112282 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.112212    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-model-cache" pod "05a9650a-d6e0-4509-a65b-4b60e98107da" (UID: "05a9650a-d6e0-4509-a65b-4b60e98107da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.112339 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.112289    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a9650a-d6e0-4509-a65b-4b60e98107da-kube-api-access-gxg4m\" (UniqueName: \"kubernetes.io/projected/05a9650a-d6e0-4509-a65b-4b60e98107da-kube-api-access-gxg4m\") pod \"05a9650a-d6e0-4509-a65b-4b60e98107da\" (UID: \"05a9650a-d6e0-4509-a65b-4b60e98107da\") "
Nov 02 02:11:28.112417 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.112391    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-home" pod "05a9650a-d6e0-4509-a65b-4b60e98107da" (UID: "05a9650a-d6e0-4509-a65b-4b60e98107da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.112654 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.112448    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-tmp-dir" pod "05a9650a-d6e0-4509-a65b-4b60e98107da" (UID: "05a9650a-d6e0-4509-a65b-4b60e98107da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.112654 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.112580    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.114306 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.114281    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a9650a-d6e0-4509-a65b-4b60e98107da-tls-certs" pod "05a9650a-d6e0-4509-a65b-4b60e98107da" (UID: "05a9650a-d6e0-4509-a65b-4b60e98107da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:28.114401 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.114359    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-dshm" pod "05a9650a-d6e0-4509-a65b-4b60e98107da" (UID: "05a9650a-d6e0-4509-a65b-4b60e98107da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:28.114548 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.114532    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a9650a-d6e0-4509-a65b-4b60e98107da-kube-api-access-gxg4m" pod "05a9650a-d6e0-4509-a65b-4b60e98107da" (UID: "05a9650a-d6e0-4509-a65b-4b60e98107da"). InnerVolumeSpecName "kube-api-access-gxg4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:28.213180 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.213140    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.213180 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.213182    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.213374 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.213196    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a9650a-d6e0-4509-a65b-4b60e98107da-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.213374 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.213210    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxg4m\" (UniqueName: \"kubernetes.io/projected/05a9650a-d6e0-4509-a65b-4b60e98107da-kube-api-access-gxg4m\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.213374 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.213228    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a9650a-d6e0-4509-a65b-4b60e98107da-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:28.753675 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.753642    4287 generic.go:356] "Generic (PLEG): container finished" podID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerID="e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a" exitCode=0
Nov 02 02:11:28.753857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.753722    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" event={"ID":"05a9650a-d6e0-4509-a65b-4b60e98107da","Type":"ContainerDied","Data":"e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a"}
Nov 02 02:11:28.753857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.753752    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s" event={"ID":"05a9650a-d6e0-4509-a65b-4b60e98107da","Type":"ContainerDied","Data":"6d1c385bc20e4d949eadeae8f5e0b861b627d068b373a2e72ab2a2bed8f6c32d"}
Nov 02 02:11:28.753857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.753759    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"
Nov 02 02:11:28.753857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.753772    4287 scope.go:122] "RemoveContainer" containerID="e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a"
Nov 02 02:11:28.762735 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.762708    4287 scope.go:122] "RemoveContainer" containerID="e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a"
Nov 02 02:11:28.762989 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:11:28.762967    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a\": container with ID starting with e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a not found: ID does not exist" containerID="e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a"
Nov 02 02:11:28.763056 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.763003    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a"} err="failed to get container status \"e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a\": rpc error: code = NotFound desc = could not find container \"e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a\": container with ID starting with e85f41b44944f1d35a63475748fb5627a5d254a4a56074c8b0c055e11d4b5e0a not found: ID does not exist"
Nov 02 02:11:28.917835 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.917781    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"]
Nov 02 02:11:28.967069 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:28.967040    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-w3-88e76d-kserve-65fc8fcbbd-m989s"]
Nov 02 02:11:30.236755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:30.236718    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" path="/var/lib/kubelet/pods/05a9650a-d6e0-4509-a65b-4b60e98107da/volumes"
Nov 02 02:11:46.038823 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:11:46.038780    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:11:48.007283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:48.007250    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"]
Nov 02 02:11:48.007691 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:48.007440    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy" containerID="cri-o://9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950" gracePeriod=30
Nov 02 02:11:48.091865 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:48.091832    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"]
Nov 02 02:11:53.251920 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.251898    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 02:11:53.363873 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.363837    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-credential-socket\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364029 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.363888    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-podinfo\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364029 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.363987    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-envoy\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364160 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364045    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-token\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-token\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364160 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364075    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-data\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364160 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364125    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-credential-socket" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:53.364160 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364149    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-kube-api-access-krfpc\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-kube-api-access-krfpc\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364360 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364180    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c19939-a073-48f8-af1e-d5695cf3327f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c19939-a073-48f8-af1e-d5695cf3327f-istiod-ca-cert\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364360 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364209    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-certs\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364360 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364260    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-socket\") pod \"d5c19939-a073-48f8-af1e-d5695cf3327f\" (UID: \"d5c19939-a073-48f8-af1e-d5695cf3327f\") "
Nov 02 02:11:53.364500 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364388    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-data" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:53.364558 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364518    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-certs" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:53.364989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364915    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-socket" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:53.364989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364959    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.364989 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364979    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.365231 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.364994    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.365284 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.365263    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c19939-a073-48f8-af1e-d5695cf3327f-istiod-ca-cert" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:53.366362 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.366336    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-podinfo" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:11:53.366471 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.366425    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-envoy" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:11:53.366841 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.366821    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-kube-api-access-krfpc" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "kube-api-access-krfpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:53.366886 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.366843    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-token" pod "d5c19939-a073-48f8-af1e-d5695cf3327f" (UID: "d5c19939-a073-48f8-af1e-d5695cf3327f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:53.466117 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.466082    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.466117 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.466112    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.466117 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.466123    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krfpc\" (UniqueName: \"kubernetes.io/projected/d5c19939-a073-48f8-af1e-d5695cf3327f-kube-api-access-krfpc\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.466365 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.466134    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c19939-a073-48f8-af1e-d5695cf3327f-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.466365 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.466143    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c19939-a073-48f8-af1e-d5695cf3327f-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.466365 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.466151    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c19939-a073-48f8-af1e-d5695cf3327f-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:53.896096 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.896058    4287 generic.go:356] "Generic (PLEG): container finished" podID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerID="9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950" exitCode=0
Nov 02 02:11:53.896295 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.896107    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"
Nov 02 02:11:53.896295 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.896133    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" event={"ID":"d5c19939-a073-48f8-af1e-d5695cf3327f","Type":"ContainerDied","Data":"9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950"}
Nov 02 02:11:53.896295 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.896185    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9" event={"ID":"d5c19939-a073-48f8-af1e-d5695cf3327f","Type":"ContainerDied","Data":"eb942eab81461ccf0ad2313aaa7a7e67c73ea14f163c4d09f3835b94799d2458"}
Nov 02 02:11:53.896295 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.896214    4287 scope.go:122] "RemoveContainer" containerID="9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950"
Nov 02 02:11:53.906077 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.906061    4287 scope.go:122] "RemoveContainer" containerID="9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950"
Nov 02 02:11:53.906320 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:11:53.906303    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950\": container with ID starting with 9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950 not found: ID does not exist" containerID="9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950"
Nov 02 02:11:53.906405 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:53.906328    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950"} err="failed to get container status \"9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950\": rpc error: code = NotFound desc = could not find container \"9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950\": container with ID starting with 9787a3fc97ea0aa9d197debe05bfcd5c474964e8fd663108e2a1e148c0427950 not found: ID does not exist"
Nov 02 02:11:54.064030 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:54.064001    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"]
Nov 02 02:11:54.113577 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:54.113544    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-gndn9"]
Nov 02 02:11:54.236967 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:54.236897    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" path="/var/lib/kubelet/pods/d5c19939-a073-48f8-af1e-d5695cf3327f/volumes"
Nov 02 02:11:59.444034 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.443990    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"]
Nov 02 02:11:59.444704 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.444206    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv" podUID="1f0d821c-ae92-41fa-975d-940af48c0730" containerName="maas-api" containerID="cri-o://238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a" gracePeriod=30
Nov 02 02:11:59.694555 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.694478    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 02:11:59.826478 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.826445    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-metrics-certs\") pod \"1f0d821c-ae92-41fa-975d-940af48c0730\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") "
Nov 02 02:11:59.826686 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.826523    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0d821c-ae92-41fa-975d-940af48c0730-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d821c-ae92-41fa-975d-940af48c0730-openshift-service-ca\") pod \"1f0d821c-ae92-41fa-975d-940af48c0730\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") "
Nov 02 02:11:59.826686 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.826567    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0d821c-ae92-41fa-975d-940af48c0730-kube-api-access-ptp6x\" (UniqueName: \"kubernetes.io/projected/1f0d821c-ae92-41fa-975d-940af48c0730-kube-api-access-ptp6x\") pod \"1f0d821c-ae92-41fa-975d-940af48c0730\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") "
Nov 02 02:11:59.826827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.826683    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls\") pod \"1f0d821c-ae92-41fa-975d-940af48c0730\" (UID: \"1f0d821c-ae92-41fa-975d-940af48c0730\") "
Nov 02 02:11:59.826896 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.826870    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0d821c-ae92-41fa-975d-940af48c0730-openshift-service-ca" pod "1f0d821c-ae92-41fa-975d-940af48c0730" (UID: "1f0d821c-ae92-41fa-975d-940af48c0730"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:11:59.827079 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.827057    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f0d821c-ae92-41fa-975d-940af48c0730-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:59.828767 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.828739    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-metrics-certs" pod "1f0d821c-ae92-41fa-975d-940af48c0730" (UID: "1f0d821c-ae92-41fa-975d-940af48c0730"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:59.828885 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.828821    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls" pod "1f0d821c-ae92-41fa-975d-940af48c0730" (UID: "1f0d821c-ae92-41fa-975d-940af48c0730"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:11:59.828885 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.828848    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0d821c-ae92-41fa-975d-940af48c0730-kube-api-access-ptp6x" pod "1f0d821c-ae92-41fa-975d-940af48c0730" (UID: "1f0d821c-ae92-41fa-975d-940af48c0730"). InnerVolumeSpecName "kube-api-access-ptp6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:11:59.927535 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.927504    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptp6x\" (UniqueName: \"kubernetes.io/projected/1f0d821c-ae92-41fa-975d-940af48c0730-kube-api-access-ptp6x\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:59.927535 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.927530    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:59.927763 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.927544    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0d821c-ae92-41fa-975d-940af48c0730-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:11:59.930585 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.930561    4287 generic.go:356] "Generic (PLEG): container finished" podID="1f0d821c-ae92-41fa-975d-940af48c0730" containerID="238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a" exitCode=0
Nov 02 02:11:59.930722 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.930639    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"
Nov 02 02:11:59.930722 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.930647    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv" event={"ID":"1f0d821c-ae92-41fa-975d-940af48c0730","Type":"ContainerDied","Data":"238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a"}
Nov 02 02:11:59.930722 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.930702    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv" event={"ID":"1f0d821c-ae92-41fa-975d-940af48c0730","Type":"ContainerDied","Data":"5943ce98fbad1b505d5eff96622fd7ba54424f0d9d9a2275be6cd5ed296b5db3"}
Nov 02 02:11:59.930859 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.930723    4287 scope.go:122] "RemoveContainer" containerID="238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a"
Nov 02 02:11:59.941548 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.941453    4287 scope.go:122] "RemoveContainer" containerID="238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a"
Nov 02 02:11:59.942311 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:11:59.941791    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a\": container with ID starting with 238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a not found: ID does not exist" containerID="238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a"
Nov 02 02:11:59.943488 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:11:59.941841    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a"} err="failed to get container status \"238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a\": rpc error: code = NotFound desc = could not find container \"238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a\": container with ID starting with 238a1d593a21759a0d51b3fc9d585e1b4c38cef3d01feb5b5edc75dc3e79243a not found: ID does not exist"
Nov 02 02:12:00.097055 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:00.097019    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"]
Nov 02 02:12:00.100973 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:00.100946    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-88e76d-58cc6d7c7b-f27fv"]
Nov 02 02:12:00.237902 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:00.237866    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0d821c-ae92-41fa-975d-940af48c0730" path="/var/lib/kubelet/pods/1f0d821c-ae92-41fa-975d-940af48c0730/volumes"
Nov 02 02:12:03.113865 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.113801    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4" podUID="3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" containerName="main" containerID="cri-o://9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db" gracePeriod=45
Nov 02 02:12:03.363550 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.363528    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:12:03.461900 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.461828    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-dshm\") pod \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") "
Nov 02 02:12:03.461900 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.461867    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-model-cache\") pod \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") "
Nov 02 02:12:03.461900 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.461897    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-kube-api-access-vlmmn\" (UniqueName: \"kubernetes.io/projected/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-kube-api-access-vlmmn\") pod \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") "
Nov 02 02:12:03.462174 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.461959    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tls-certs\") pod \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") "
Nov 02 02:12:03.462174 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.461982    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-home\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-home\") pod \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") "
Nov 02 02:12:03.462174 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.462044    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tmp-dir\") pod \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\" (UID: \"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29\") "
Nov 02 02:12:03.462174 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.462122    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-model-cache" pod "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" (UID: "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:03.462383 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.462278    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-home" pod "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" (UID: "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:03.462383 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.462292    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tmp-dir" pod "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" (UID: "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:03.462472 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.462434    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:03.462472 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.462463    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:03.462539 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.462472    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:03.464349 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.464327    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-kube-api-access-vlmmn" pod "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" (UID: "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29"). InnerVolumeSpecName "kube-api-access-vlmmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:03.464634 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.464591    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-dshm" pod "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" (UID: "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:03.464729 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.464627    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tls-certs" pod "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" (UID: "3bf5bc1f-48d5-4811-bd1a-c3939e31cb29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:03.563753 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.563721    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:03.563753 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.563749    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlmmn\" (UniqueName: \"kubernetes.io/projected/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-kube-api-access-vlmmn\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:03.563753 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.563760    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:03.956341 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.956309    4287 generic.go:356] "Generic (PLEG): container finished" podID="3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" containerID="9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db" exitCode=0
Nov 02 02:12:03.956527 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.956352    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"
Nov 02 02:12:03.956527 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.956406    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4" event={"ID":"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29","Type":"ContainerDied","Data":"9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db"}
Nov 02 02:12:03.956527 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.956450    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4" event={"ID":"3bf5bc1f-48d5-4811-bd1a-c3939e31cb29","Type":"ContainerDied","Data":"149593db865e24802524de1cf1eca91d3b67249b62568a6ba23577e8ec17824c"}
Nov 02 02:12:03.956527 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.956473    4287 scope.go:122] "RemoveContainer" containerID="9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db"
Nov 02 02:12:03.966512 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.966498    4287 scope.go:122] "RemoveContainer" containerID="9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db"
Nov 02 02:12:03.966830 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:12:03.966806    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db\": container with ID starting with 9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db not found: ID does not exist" containerID="9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db"
Nov 02 02:12:03.966904 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:03.966840    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db"} err="failed to get container status \"9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db\": rpc error: code = NotFound desc = could not find container \"9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db\": container with ID starting with 9ab975ad8bc5c57f4acb55cd48c157d077ce75cbcc368af1b8156a934df513db not found: ID does not exist"
Nov 02 02:12:04.143581 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:04.143544    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"]
Nov 02 02:12:04.178527 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:04.178488    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d0fdb9/test-model-d0fdb9-kserve-7bc96cd98f-tnlr4"]
Nov 02 02:12:04.237204 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:04.237138    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" path="/var/lib/kubelet/pods/3bf5bc1f-48d5-4811-bd1a-c3939e31cb29/volumes"
Nov 02 02:12:08.602616 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:08.602577    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"]
Nov 02 02:12:08.603064 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:08.602850    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy" containerID="cri-o://0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702" gracePeriod=30
Nov 02 02:12:09.761713 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:09.761681    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"]
Nov 02 02:12:13.862869 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.862842    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 02:12:13.957119 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957059    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-podinfo\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957119 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957098    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-kube-api-access-fm4r2\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-kube-api-access-fm4r2\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957298 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957125    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-socket\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957298 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957147    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-certs\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957298 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957179    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-credential-socket\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957298 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957228    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-data\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957298 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957255    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f9d676-48a6-4382-9ed7-0e9992dc925a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f9d676-48a6-4382-9ed7-0e9992dc925a-istiod-ca-cert\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957298 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957293    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-token\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-token\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957643 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957325    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-envoy\") pod \"45f9d676-48a6-4382-9ed7-0e9992dc925a\" (UID: \"45f9d676-48a6-4382-9ed7-0e9992dc925a\") "
Nov 02 02:12:13.957643 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957436    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-socket" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:13.957643 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957507    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-certs" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:13.957643 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957575    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-data" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:13.957861 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957776    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:13.957861 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957799    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:13.957861 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957827    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:13.957861 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957766    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-credential-socket" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:13.958006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.957862    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f9d676-48a6-4382-9ed7-0e9992dc925a-istiod-ca-cert" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:12:13.959910 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.959883    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-podinfo" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:12:13.959910 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.959897    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-kube-api-access-fm4r2" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "kube-api-access-fm4r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:13.960090 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.960074    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-envoy" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:13.960090 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:13.960077    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-token" pod "45f9d676-48a6-4382-9ed7-0e9992dc925a" (UID: "45f9d676-48a6-4382-9ed7-0e9992dc925a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:14.013417 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.013386    4287 generic.go:356] "Generic (PLEG): container finished" podID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerID="0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702" exitCode=0
Nov 02 02:12:14.013539 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.013436    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"
Nov 02 02:12:14.013539 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.013476    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" event={"ID":"45f9d676-48a6-4382-9ed7-0e9992dc925a","Type":"ContainerDied","Data":"0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702"}
Nov 02 02:12:14.013630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.013560    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n" event={"ID":"45f9d676-48a6-4382-9ed7-0e9992dc925a","Type":"ContainerDied","Data":"c329bd8e895acd6de56ee6ce3d595ffa4165234ffbf2b03ae99b3a0b02bd11c3"}
Nov 02 02:12:14.013630 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.013584    4287 scope.go:122] "RemoveContainer" containerID="0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702"
Nov 02 02:12:14.023559 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.023537    4287 scope.go:122] "RemoveContainer" containerID="0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702"
Nov 02 02:12:14.023868 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:12:14.023848    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702\": container with ID starting with 0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702 not found: ID does not exist" containerID="0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702"
Nov 02 02:12:14.023938 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.023875    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702"} err="failed to get container status \"0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702\": rpc error: code = NotFound desc = could not find container \"0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702\": container with ID starting with 0184e6e701cc71e9cf0b551bdd357adca44910442094c5322984717ed6d3d702 not found: ID does not exist"
Nov 02 02:12:14.058946 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.058918    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f9d676-48a6-4382-9ed7-0e9992dc925a-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.058946 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.058943    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.059102 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.058953    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.059102 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.058963    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f9d676-48a6-4382-9ed7-0e9992dc925a-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.059102 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.058973    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm4r2\" (UniqueName: \"kubernetes.io/projected/45f9d676-48a6-4382-9ed7-0e9992dc925a-kube-api-access-fm4r2\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.059102 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.058983    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f9d676-48a6-4382-9ed7-0e9992dc925a-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:14.145494 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.145463    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"]
Nov 02 02:12:14.151036 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.151012    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-7zn7n"]
Nov 02 02:12:14.237063 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:14.236998    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" path="/var/lib/kubelet/pods/45f9d676-48a6-4382-9ed7-0e9992dc925a/volumes"
Nov 02 02:12:16.040163 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:12:16.040123    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:12:24.781830 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:24.781783    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6" podUID="be0c997b-eb96-467b-a3e2-84fd7450db9a" containerName="main" containerID="cri-o://f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0" gracePeriod=45
Nov 02 02:12:24.837327 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:24.837287    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6" podUID="be0c997b-eb96-467b-a3e2-84fd7450db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": dial tcp 10.133.0.101:8000: connect: connection refused"
Nov 02 02:12:25.033810 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.033747    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:12:25.072489 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.072448    4287 generic.go:356] "Generic (PLEG): container finished" podID="be0c997b-eb96-467b-a3e2-84fd7450db9a" containerID="f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0" exitCode=0
Nov 02 02:12:25.072674 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.072493    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"
Nov 02 02:12:25.072674 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.072539    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6" event={"ID":"be0c997b-eb96-467b-a3e2-84fd7450db9a","Type":"ContainerDied","Data":"f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0"}
Nov 02 02:12:25.072674 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.072580    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6" event={"ID":"be0c997b-eb96-467b-a3e2-84fd7450db9a","Type":"ContainerDied","Data":"b17c10dc352955c49b21dc4fc551a914f8d988b5176fe7d3eae571e65922e278"}
Nov 02 02:12:25.072674 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.072599    4287 scope.go:122] "RemoveContainer" containerID="f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0"
Nov 02 02:12:25.081168 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.081152    4287 scope.go:122] "RemoveContainer" containerID="f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0"
Nov 02 02:12:25.081439 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:12:25.081423    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0\": container with ID starting with f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0 not found: ID does not exist" containerID="f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0"
Nov 02 02:12:25.081485 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.081455    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0"} err="failed to get container status \"f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0\": rpc error: code = NotFound desc = could not find container \"f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0\": container with ID starting with f1c1275daabd94955d066ed03ceaf97a599f9f69d5f7f6a7722859a956cac0f0 not found: ID does not exist"
Nov 02 02:12:25.173216 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173185    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-home\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-home\") pod \"be0c997b-eb96-467b-a3e2-84fd7450db9a\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") "
Nov 02 02:12:25.173392 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173249    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-dshm\") pod \"be0c997b-eb96-467b-a3e2-84fd7450db9a\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") "
Nov 02 02:12:25.173392 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173283    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-tmp-dir\") pod \"be0c997b-eb96-467b-a3e2-84fd7450db9a\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") "
Nov 02 02:12:25.173392 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173320    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be0c997b-eb96-467b-a3e2-84fd7450db9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/be0c997b-eb96-467b-a3e2-84fd7450db9a-tls-certs\") pod \"be0c997b-eb96-467b-a3e2-84fd7450db9a\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") "
Nov 02 02:12:25.173392 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173377    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-model-cache\") pod \"be0c997b-eb96-467b-a3e2-84fd7450db9a\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") "
Nov 02 02:12:25.173739 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173513    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-home" pod "be0c997b-eb96-467b-a3e2-84fd7450db9a" (UID: "be0c997b-eb96-467b-a3e2-84fd7450db9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:25.173739 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173527    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-tmp-dir" pod "be0c997b-eb96-467b-a3e2-84fd7450db9a" (UID: "be0c997b-eb96-467b-a3e2-84fd7450db9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:25.173739 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173563    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be0c997b-eb96-467b-a3e2-84fd7450db9a-kube-api-access-xwdgl\" (UniqueName: \"kubernetes.io/projected/be0c997b-eb96-467b-a3e2-84fd7450db9a-kube-api-access-xwdgl\") pod \"be0c997b-eb96-467b-a3e2-84fd7450db9a\" (UID: \"be0c997b-eb96-467b-a3e2-84fd7450db9a\") "
Nov 02 02:12:25.173739 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173728    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-model-cache" pod "be0c997b-eb96-467b-a3e2-84fd7450db9a" (UID: "be0c997b-eb96-467b-a3e2-84fd7450db9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:25.174001 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.173981    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:25.174063 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.174009    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:25.174063 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.174024    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:25.175839 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.175812    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0c997b-eb96-467b-a3e2-84fd7450db9a-kube-api-access-xwdgl" pod "be0c997b-eb96-467b-a3e2-84fd7450db9a" (UID: "be0c997b-eb96-467b-a3e2-84fd7450db9a"). InnerVolumeSpecName "kube-api-access-xwdgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:12:25.176173 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.176147    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-dshm" pod "be0c997b-eb96-467b-a3e2-84fd7450db9a" (UID: "be0c997b-eb96-467b-a3e2-84fd7450db9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:12:25.176264 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.176165    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0c997b-eb96-467b-a3e2-84fd7450db9a-tls-certs" pod "be0c997b-eb96-467b-a3e2-84fd7450db9a" (UID: "be0c997b-eb96-467b-a3e2-84fd7450db9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:12:25.275218 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.275185    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0c997b-eb96-467b-a3e2-84fd7450db9a-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:25.275218 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.275214    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0c997b-eb96-467b-a3e2-84fd7450db9a-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:25.275407 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.275227    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwdgl\" (UniqueName: \"kubernetes.io/projected/be0c997b-eb96-467b-a3e2-84fd7450db9a-kube-api-access-xwdgl\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:12:25.495973 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.495931    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"]
Nov 02 02:12:25.499653 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:25.499627    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-361fcc/test-model-361fcc-kserve-7fbdfd6d55-z6qb6"]
Nov 02 02:12:26.239026 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:12:26.238916    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be0c997b-eb96-467b-a3e2-84fd7450db9a" path="/var/lib/kubelet/pods/be0c997b-eb96-467b-a3e2-84fd7450db9a/volumes"
Nov 02 02:12:46.037134 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:12:46.037091    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:13:16.038596 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:13:16.038555    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:13:46.037889 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:13:46.037853    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:14:07.101215 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.101182    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"]
Nov 02 02:14:07.101766 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.101376    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b" podUID="a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" containerName="maas-api" containerID="cri-o://17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09" gracePeriod=30
Nov 02 02:14:07.376433 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.376410    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 02:14:07.509194 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.509158    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-kube-api-access-zxh5x\" (UniqueName: \"kubernetes.io/projected/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-kube-api-access-zxh5x\") pod \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") "
Nov 02 02:14:07.509194 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.509200    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-openshift-service-ca\") pod \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") "
Nov 02 02:14:07.509454 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.509232    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-metrics-certs\") pod \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") "
Nov 02 02:14:07.509454 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.509343    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls\") pod \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\" (UID: \"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b\") "
Nov 02 02:14:07.509590 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.509563    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-openshift-service-ca" pod "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" (UID: "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:07.509818 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.509792    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:07.511527 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.511503    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls" pod "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" (UID: "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:07.511706 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.511566    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-metrics-certs" pod "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" (UID: "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:07.511706 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.511578    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-kube-api-access-zxh5x" pod "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" (UID: "a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b"). InnerVolumeSpecName "kube-api-access-zxh5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:07.585084 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.585051    4287 generic.go:356] "Generic (PLEG): container finished" podID="a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" containerID="17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09" exitCode=0
Nov 02 02:14:07.585249 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.585094    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"
Nov 02 02:14:07.585249 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.585143    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b" event={"ID":"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b","Type":"ContainerDied","Data":"17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09"}
Nov 02 02:14:07.585249 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.585194    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b" event={"ID":"a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b","Type":"ContainerDied","Data":"c915f07a07bea1b9635d091529a4db4acdd9951ac3114d926e2c1fbd6c317c4d"}
Nov 02 02:14:07.585249 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.585216    4287 scope.go:122] "RemoveContainer" containerID="17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09"
Nov 02 02:14:07.595314 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.595300    4287 scope.go:122] "RemoveContainer" containerID="17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09"
Nov 02 02:14:07.595595 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:14:07.595577    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09\": container with ID starting with 17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09 not found: ID does not exist" containerID="17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09"
Nov 02 02:14:07.595671 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.595619    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09"} err="failed to get container status \"17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09\": rpc error: code = NotFound desc = could not find container \"17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09\": container with ID starting with 17e7ea0d5f8a2de2b770507806d8f1ea41019d0121f41f20d3cafc570fc86b09 not found: ID does not exist"
Nov 02 02:14:07.611250 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.611165    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxh5x\" (UniqueName: \"kubernetes.io/projected/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-kube-api-access-zxh5x\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:07.611250 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.611204    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:07.611250 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.611216    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:07.748645 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.748597    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"]
Nov 02 02:14:07.752539 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:07.752515    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-39ceb2-d8b97b644-jcf8b"]
Nov 02 02:14:08.236424 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:08.236390    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" path="/var/lib/kubelet/pods/a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b/volumes"
Nov 02 02:14:08.616122 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:08.616087    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"]
Nov 02 02:14:08.616337 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:08.616312    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy" containerID="cri-o://add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf" gracePeriod=30
Nov 02 02:14:13.867484 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.867459    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:14:13.971495 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971430    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-istio-token\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-istio-token\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.971495 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971475    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-socket\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.971711 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971515    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-data\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.971711 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971640    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-envoy\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.971827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971713    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-credential-socket\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.971827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971756    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-kube-api-access-xrjb9\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-kube-api-access-xrjb9\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.971827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971765    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-data" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:13.971986 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971793    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bd95e2-4708-48e4-831d-05c676377146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bd95e2-4708-48e4-831d-05c676377146-istiod-ca-cert\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.971986 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971830    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-socket" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:13.971986 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971937    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-certs\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.971986 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971944    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-credential-socket" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:13.972198 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.971987    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6bd95e2-4708-48e4-831d-05c676377146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bd95e2-4708-48e4-831d-05c676377146-istio-podinfo\") pod \"b6bd95e2-4708-48e4-831d-05c676377146\" (UID: \"b6bd95e2-4708-48e4-831d-05c676377146\") "
Nov 02 02:14:13.972253 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.972193    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bd95e2-4708-48e4-831d-05c676377146-istiod-ca-cert" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:13.972343 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.972320    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-certs" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:13.972407 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.972342    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:13.972407 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.972362    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:13.972407 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.972379    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6bd95e2-4708-48e4-831d-05c676377146-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:13.972407 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.972392    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:13.973813 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.973785    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-istio-token" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:13.973934 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.973904    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-kube-api-access-xrjb9" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "kube-api-access-xrjb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:13.974028 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.974010    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-envoy" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:13.974285 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:13.974261    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6bd95e2-4708-48e4-831d-05c676377146-istio-podinfo" pod "b6bd95e2-4708-48e4-831d-05c676377146" (UID: "b6bd95e2-4708-48e4-831d-05c676377146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:14:14.073333 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.073300    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjb9\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-kube-api-access-xrjb9\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:14.073333 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.073328    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:14.073520 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.073341    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6bd95e2-4708-48e4-831d-05c676377146-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:14.073520 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.073353    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6bd95e2-4708-48e4-831d-05c676377146-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:14.073520 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.073366    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6bd95e2-4708-48e4-831d-05c676377146-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:14.621265 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.621224    4287 generic.go:356] "Generic (PLEG): container finished" podID="b6bd95e2-4708-48e4-831d-05c676377146" containerID="add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf" exitCode=0
Nov 02 02:14:14.621514 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.621274    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"
Nov 02 02:14:14.621514 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.621318    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" event={"ID":"b6bd95e2-4708-48e4-831d-05c676377146","Type":"ContainerDied","Data":"add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf"}
Nov 02 02:14:14.621514 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.621374    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk" event={"ID":"b6bd95e2-4708-48e4-831d-05c676377146","Type":"ContainerDied","Data":"05827279d0e1e8ec0bf0ee491fe7d564c681b2c24ae4851b5a88b4e202940a10"}
Nov 02 02:14:14.621514 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.621396    4287 scope.go:122] "RemoveContainer" containerID="add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf"
Nov 02 02:14:14.630418 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.630393    4287 scope.go:122] "RemoveContainer" containerID="add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf"
Nov 02 02:14:14.630706 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:14:14.630686    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf\": container with ID starting with add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf not found: ID does not exist" containerID="add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf"
Nov 02 02:14:14.630770 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.630720    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf"} err="failed to get container status \"add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf\": rpc error: code = NotFound desc = could not find container \"add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf\": container with ID starting with add80063133f9f29780870a594195a02bb361da0338ff50af54116bb5b6fbcaf not found: ID does not exist"
Nov 02 02:14:14.786139 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.786112    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"]
Nov 02 02:14:14.790163 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:14.790142    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxwdk"]
Nov 02 02:14:16.037704 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:14:16.037671    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:14:16.236253 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:16.236219    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bd95e2-4708-48e4-831d-05c676377146" path="/var/lib/kubelet/pods/b6bd95e2-4708-48e4-831d-05c676377146/volumes"
Nov 02 02:14:21.415313 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:21.415278    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"]
Nov 02 02:14:36.435567 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.435511    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" containerID="cri-o://550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4" gracePeriod=45
Nov 02 02:14:36.438791 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.438755    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused"
Nov 02 02:14:36.682803 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.682777    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 02:14:36.737399 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.737325    4287 generic.go:356] "Generic (PLEG): container finished" podID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerID="550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4" exitCode=0
Nov 02 02:14:36.737399 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.737368    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"
Nov 02 02:14:36.737569 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.737404    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" event={"ID":"6a1a976f-fa19-4dad-a28c-5a8039ff41ae","Type":"ContainerDied","Data":"550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4"}
Nov 02 02:14:36.737569 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.737440    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq" event={"ID":"6a1a976f-fa19-4dad-a28c-5a8039ff41ae","Type":"ContainerDied","Data":"bfb0ce9c04490394ea4d266d28fa558eab74eac71cadcaf406d1e5b8f0f04445"}
Nov 02 02:14:36.737569 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.737457    4287 scope.go:122] "RemoveContainer" containerID="550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4"
Nov 02 02:14:36.745983 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.745963    4287 scope.go:122] "RemoveContainer" containerID="550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4"
Nov 02 02:14:36.746256 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:14:36.746237    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4\": container with ID starting with 550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4 not found: ID does not exist" containerID="550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4"
Nov 02 02:14:36.746329 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.746267    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4"} err="failed to get container status \"550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4\": rpc error: code = NotFound desc = could not find container \"550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4\": container with ID starting with 550ff02f61c4d34dbbc0c645c49fb3ad431e8df02752043f83922a940d4423c4 not found: ID does not exist"
Nov 02 02:14:36.774200 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774167    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-model-cache\") pod \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") "
Nov 02 02:14:36.774321 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774208    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-kube-api-access-w2zxz\" (UniqueName: \"kubernetes.io/projected/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-kube-api-access-w2zxz\") pod \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") "
Nov 02 02:14:36.774321 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774240    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-home\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-home\") pod \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") "
Nov 02 02:14:36.774321 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774277    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tls-certs\") pod \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") "
Nov 02 02:14:36.774435 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774353    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tmp-dir\") pod \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") "
Nov 02 02:14:36.774435 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774390    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-dshm\") pod \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\" (UID: \"6a1a976f-fa19-4dad-a28c-5a8039ff41ae\") "
Nov 02 02:14:36.774552 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774441    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-model-cache" pod "6a1a976f-fa19-4dad-a28c-5a8039ff41ae" (UID: "6a1a976f-fa19-4dad-a28c-5a8039ff41ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:36.774638 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774559    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-home" pod "6a1a976f-fa19-4dad-a28c-5a8039ff41ae" (UID: "6a1a976f-fa19-4dad-a28c-5a8039ff41ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:36.774694 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774674    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tmp-dir" pod "6a1a976f-fa19-4dad-a28c-5a8039ff41ae" (UID: "6a1a976f-fa19-4dad-a28c-5a8039ff41ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:36.774801 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774784    4287 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tmp-dir\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.774857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774807    4287 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-model-cache\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.774857 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.774824    4287 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-home\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.776736 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.776707    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-dshm" pod "6a1a976f-fa19-4dad-a28c-5a8039ff41ae" (UID: "6a1a976f-fa19-4dad-a28c-5a8039ff41ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:36.776834 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.776812    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tls-certs" pod "6a1a976f-fa19-4dad-a28c-5a8039ff41ae" (UID: "6a1a976f-fa19-4dad-a28c-5a8039ff41ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:36.776870 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.776856    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-kube-api-access-w2zxz" pod "6a1a976f-fa19-4dad-a28c-5a8039ff41ae" (UID: "6a1a976f-fa19-4dad-a28c-5a8039ff41ae"). InnerVolumeSpecName "kube-api-access-w2zxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:36.877243 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.877211    4287 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-tls-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.877490 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.877462    4287 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-dshm\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:36.877629 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:36.877496    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zxz\" (UniqueName: \"kubernetes.io/projected/6a1a976f-fa19-4dad-a28c-5a8039ff41ae-kube-api-access-w2zxz\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:37.158890 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:37.158857    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"]
Nov 02 02:14:37.165006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:37.164973    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/distinct-w5-e06ea2-kserve-76966cdcb7-kflcq"]
Nov 02 02:14:38.236203 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:38.236169    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" path="/var/lib/kubelet/pods/6a1a976f-fa19-4dad-a28c-5a8039ff41ae/volumes"
Nov 02 02:14:43.074525 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.074488    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"]
Nov 02 02:14:43.074994 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.074733    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55" podUID="e7fe6eac-4558-4b31-bf54-c7b18365f915" containerName="maas-api" containerID="cri-o://562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f" gracePeriod=30
Nov 02 02:14:43.341690 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.341600    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 02:14:43.441766 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.441737    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7fe6eac-4558-4b31-bf54-c7b18365f915-kube-api-access-n7fpd\" (UniqueName: \"kubernetes.io/projected/e7fe6eac-4558-4b31-bf54-c7b18365f915-kube-api-access-n7fpd\") pod \"e7fe6eac-4558-4b31-bf54-c7b18365f915\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") "
Nov 02 02:14:43.441959 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.441782    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-metrics-certs\") pod \"e7fe6eac-4558-4b31-bf54-c7b18365f915\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") "
Nov 02 02:14:43.441959 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.441849    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7fe6eac-4558-4b31-bf54-c7b18365f915-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fe6eac-4558-4b31-bf54-c7b18365f915-openshift-service-ca\") pod \"e7fe6eac-4558-4b31-bf54-c7b18365f915\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") "
Nov 02 02:14:43.441959 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.441881    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls\") pod \"e7fe6eac-4558-4b31-bf54-c7b18365f915\" (UID: \"e7fe6eac-4558-4b31-bf54-c7b18365f915\") "
Nov 02 02:14:43.442233 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.442201    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7fe6eac-4558-4b31-bf54-c7b18365f915-openshift-service-ca" pod "e7fe6eac-4558-4b31-bf54-c7b18365f915" (UID: "e7fe6eac-4558-4b31-bf54-c7b18365f915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:43.444316 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.444292    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls" pod "e7fe6eac-4558-4b31-bf54-c7b18365f915" (UID: "e7fe6eac-4558-4b31-bf54-c7b18365f915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:43.444494 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.444342    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-metrics-certs" pod "e7fe6eac-4558-4b31-bf54-c7b18365f915" (UID: "e7fe6eac-4558-4b31-bf54-c7b18365f915"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:14:43.446586 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.444535    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7fe6eac-4558-4b31-bf54-c7b18365f915-kube-api-access-n7fpd" pod "e7fe6eac-4558-4b31-bf54-c7b18365f915" (UID: "e7fe6eac-4558-4b31-bf54-c7b18365f915"). InnerVolumeSpecName "kube-api-access-n7fpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:43.543349 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.543320    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7fpd\" (UniqueName: \"kubernetes.io/projected/e7fe6eac-4558-4b31-bf54-c7b18365f915-kube-api-access-n7fpd\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:43.543349 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.543344    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:43.543349 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.543355    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7fe6eac-4558-4b31-bf54-c7b18365f915-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:43.543571 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.543364    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7fe6eac-4558-4b31-bf54-c7b18365f915-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:43.775278 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.775238    4287 generic.go:356] "Generic (PLEG): container finished" podID="e7fe6eac-4558-4b31-bf54-c7b18365f915" containerID="562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f" exitCode=0
Nov 02 02:14:43.775476 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.775282    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"
Nov 02 02:14:43.775476 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.775301    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55" event={"ID":"e7fe6eac-4558-4b31-bf54-c7b18365f915","Type":"ContainerDied","Data":"562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f"}
Nov 02 02:14:43.775476 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.775372    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55" event={"ID":"e7fe6eac-4558-4b31-bf54-c7b18365f915","Type":"ContainerDied","Data":"2e2930810f0b4bdb0e6c7a8c76cda94911bfc958d0d065f47d4f36a0d370ae7b"}
Nov 02 02:14:43.775476 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.775395    4287 scope.go:122] "RemoveContainer" containerID="562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f"
Nov 02 02:14:43.787330 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.787311    4287 scope.go:122] "RemoveContainer" containerID="562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f"
Nov 02 02:14:43.787590 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:14:43.787569    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f\": container with ID starting with 562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f not found: ID does not exist" containerID="562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f"
Nov 02 02:14:43.787663 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.787600    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f"} err="failed to get container status \"562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f\": rpc error: code = NotFound desc = could not find container \"562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f\": container with ID starting with 562b4116f3cafb6d90174825e5413e7ab87a08089d589fa94ccf20b51913b44f not found: ID does not exist"
Nov 02 02:14:43.941306 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.941271    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"]
Nov 02 02:14:43.993411 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:43.993366    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e06ea2-6bbfb9795f-mzz55"]
Nov 02 02:14:44.238753 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:44.238663    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7fe6eac-4558-4b31-bf54-c7b18365f915" path="/var/lib/kubelet/pods/e7fe6eac-4558-4b31-bf54-c7b18365f915/volumes"
Nov 02 02:14:46.037515 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:14:46.037484    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:14:52.208962 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:52.208929    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"]
Nov 02 02:14:52.209589 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:52.209563    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy" containerID="cri-o://8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2" gracePeriod=30
Nov 02 02:14:52.654880 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:52.654855    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:14:52.668946 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:52.668920    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:14:57.457538 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.457508    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 02:14:57.569261 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569228    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-envoy\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569461 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569269    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70414325-f032-436c-9bb1-951077b61520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70414325-f032-436c-9bb1-951077b61520-istiod-ca-cert\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569461 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569323    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-credential-socket\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569461 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569351    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-certs\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569461 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569371    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-data\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569461 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569395    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-socket\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569461 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569454    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70414325-f032-436c-9bb1-951077b61520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70414325-f032-436c-9bb1-951077b61520-istio-podinfo\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569836 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569522    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-istio-token\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-istio-token\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569836 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569583    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-kube-api-access-t5lsc\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-kube-api-access-t5lsc\") pod \"70414325-f032-436c-9bb1-951077b61520\" (UID: \"70414325-f032-436c-9bb1-951077b61520\") "
Nov 02 02:14:57.569836 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569711    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-credential-socket" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:57.569836 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569707    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-data" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:57.569836 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569787    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-socket" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:57.570283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569836    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70414325-f032-436c-9bb1-951077b61520-istiod-ca-cert" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:14:57.570283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569948    4287 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70414325-f032-436c-9bb1-951077b61520-istiod-ca-cert\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.570283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569964    4287 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-credential-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.570283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569974    4287 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-data\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.570283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.569984    4287 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-socket\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.570283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.570007    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-certs" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:57.571978 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.571950    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-envoy" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 02:14:57.571978 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.571955    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70414325-f032-436c-9bb1-951077b61520-istio-podinfo" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 02:14:57.572124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.571998    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-istio-token" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:57.572124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.572021    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-kube-api-access-t5lsc" pod "70414325-f032-436c-9bb1-951077b61520" (UID: "70414325-f032-436c-9bb1-951077b61520"). InnerVolumeSpecName "kube-api-access-t5lsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:14:57.670763 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.670727    4287 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70414325-f032-436c-9bb1-951077b61520-istio-podinfo\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.670763 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.670757    4287 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-istio-token\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.670763 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.670767    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5lsc\" (UniqueName: \"kubernetes.io/projected/70414325-f032-436c-9bb1-951077b61520-kube-api-access-t5lsc\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.671003 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.670777    4287 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-istio-envoy\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.671003 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.670786    4287 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70414325-f032-436c-9bb1-951077b61520-workload-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:14:57.850785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.850694    4287 generic.go:356] "Generic (PLEG): container finished" podID="70414325-f032-436c-9bb1-951077b61520" containerID="8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2" exitCode=0
Nov 02 02:14:57.850785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.850735    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"
Nov 02 02:14:57.850785 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.850769    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" event={"ID":"70414325-f032-436c-9bb1-951077b61520","Type":"ContainerDied","Data":"8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2"}
Nov 02 02:14:57.850999 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.850833    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7" event={"ID":"70414325-f032-436c-9bb1-951077b61520","Type":"ContainerDied","Data":"d98f4a1585f9e38eb0478bf840cf59ba77064f3bb58b3e238220c04adb850c23"}
Nov 02 02:14:57.850999 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.850860    4287 scope.go:122] "RemoveContainer" containerID="8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2"
Nov 02 02:14:57.861243 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.861218    4287 scope.go:122] "RemoveContainer" containerID="8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2"
Nov 02 02:14:57.861566 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:14:57.861548    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2\": container with ID starting with 8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2 not found: ID does not exist" containerID="8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2"
Nov 02 02:14:57.861657 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:57.861576    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2"} err="failed to get container status \"8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2\": rpc error: code = NotFound desc = could not find container \"8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2\": container with ID starting with 8f678e40d53242073c2ef493e708576dd8edf750958b9934b1b1df7af7eb19d2 not found: ID does not exist"
Nov 02 02:14:58.015824 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:58.015783    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"]
Nov 02 02:14:58.065444 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:58.065407    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-ncnx7"]
Nov 02 02:14:58.236080 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:14:58.236004    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70414325-f032-436c-9bb1-951077b61520" path="/var/lib/kubelet/pods/70414325-f032-436c-9bb1-951077b61520/volumes"
Nov 02 02:15:16.037916 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:15:16.037875    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:15:46.037045 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:15:46.037003    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:16:16.038628 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:16:16.038575    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:16:39.831419 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831341    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"]
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831699    4287 cpu_manager.go:414] "containerMap: removing container" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831712    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831729    4287 cpu_manager.go:414] "containerMap: removing container" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831734    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831743    4287 cpu_manager.go:414] "containerMap: removing container" podUID="1f0d821c-ae92-41fa-975d-940af48c0730" containerName="maas-api"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831748    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0d821c-ae92-41fa-975d-940af48c0730" containerName="maas-api"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831754    4287 cpu_manager.go:414] "containerMap: removing container" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831759    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831766    4287 cpu_manager.go:414] "containerMap: removing container" podUID="3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" containerName="main"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831770    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" containerName="main"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831777    4287 cpu_manager.go:414] "containerMap: removing container" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831781    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831789    4287 cpu_manager.go:414] "containerMap: removing container" podUID="d32a2a55-b4ef-4548-87e5-03df6bc720e6" containerName="maas-api"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831793    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32a2a55-b4ef-4548-87e5-03df6bc720e6" containerName="maas-api"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831800    4287 cpu_manager.go:414] "containerMap: removing container" podUID="a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" containerName="maas-api"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831804    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" containerName="maas-api"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831810    4287 cpu_manager.go:414] "containerMap: removing container" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831815    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831822    4287 cpu_manager.go:414] "containerMap: removing container" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831826    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831834    4287 cpu_manager.go:414] "containerMap: removing container" podUID="e7fe6eac-4558-4b31-bf54-c7b18365f915" containerName="maas-api"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831839    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7fe6eac-4558-4b31-bf54-c7b18365f915" containerName="maas-api"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831847    4287 cpu_manager.go:414] "containerMap: removing container" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main"
Nov 02 02:16:39.831845 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831852    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831863    4287 cpu_manager.go:414] "containerMap: removing container" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831868    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831875    4287 cpu_manager.go:414] "containerMap: removing container" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831879    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831888    4287 cpu_manager.go:414] "containerMap: removing container" podUID="4b496963-737d-4d4b-b618-ad31f1305896" containerName="maas-api"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831892    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b496963-737d-4d4b-b618-ad31f1305896" containerName="maas-api"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831900    4287 cpu_manager.go:414] "containerMap: removing container" podUID="be0c997b-eb96-467b-a3e2-84fd7450db9a" containerName="main"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831904    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be0c997b-eb96-467b-a3e2-84fd7450db9a" containerName="main"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831955    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831961    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f9d676-48a6-4382-9ed7-0e9992dc925a" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831969    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32a2a55-b4ef-4548-87e5-03df6bc720e6" containerName="maas-api"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831974    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b496963-737d-4d4b-b618-ad31f1305896" containerName="maas-api"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831981    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47c8503-f9e9-4dba-a9c6-4eb6ec1eb56b" containerName="maas-api"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831987    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bd95e2-4708-48e4-831d-05c676377146" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831992    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a9650a-d6e0-4509-a65b-4b60e98107da" containerName="main"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.831997    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="be0c997b-eb96-467b-a3e2-84fd7450db9a" containerName="main"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832002    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832006    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7fe6eac-4558-4b31-bf54-c7b18365f915" containerName="maas-api"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832012    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1a976f-fa19-4dad-a28c-5a8039ff41ae" containerName="main"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832016    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="351cb013-4b1d-4d42-b1d8-9e45632c86d5" containerName="main"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832021    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c19939-a073-48f8-af1e-d5695cf3327f" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832026    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0d821c-ae92-41fa-975d-940af48c0730" containerName="maas-api"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832031    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="70414325-f032-436c-9bb1-951077b61520" containerName="istio-proxy"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832036    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf5bc1f-48d5-4811-bd1a-c3939e31cb29" containerName="main"
Nov 02 02:16:39.832541 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.832041    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3b16cf-2480-4289-ab1a-e429adb66645" containerName="istio-proxy"
Nov 02 02:16:39.835092 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:39.835077    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006039 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006002    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006044    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006123    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006144    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006165    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006229 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006194    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006408 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006301    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d42v9\" (UniqueName: \"kubernetes.io/projected/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-kube-api-access-d42v9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006408 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006326    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.006408 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.006392    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.047310 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.047273    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"]
Nov 02 02:16:40.107432 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107358    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.107432 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107394    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.107432 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107420    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.107755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107438    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.107755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107455    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.107755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107482    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.107755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107518    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d42v9\" (UniqueName: \"kubernetes.io/projected/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-kube-api-access-d42v9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.107755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107545    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.107755 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107601    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.108079 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.107990    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.108079 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.108055    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.108177 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.108098    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.108288 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.108263    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.108424 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.108406    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.109900 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.109880    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.110124 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.110105    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.167996 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.167964    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.168149 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.168032    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d42v9\" (UniqueName: \"kubernetes.io/projected/e26462c2-dfb3-4760-9f7b-ee4bd50c2e63-kube-api-access-d42v9\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-kmgsf\" (UID: \"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.447027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.446937    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:40.826578 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.825260    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"]
Nov 02 02:16:40.831272 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.831233    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:16:40.831387 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.831370    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:16:40.831467 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:40.831452    4287 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 02 02:16:41.379917 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:41.379880    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" event={"ID":"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63","Type":"ContainerStarted","Data":"6a5509b78cc77d354f203a01164f1521c741372e342eac96b88d39ad49fff6ff"}
Nov 02 02:16:41.380097 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:41.379943    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" event={"ID":"e26462c2-dfb3-4760-9f7b-ee4bd50c2e63","Type":"ContainerStarted","Data":"e5e853d46146e2c3d35b4d566a0c06f5f542eeb8ec83b7985373eca02f2e99a5"}
Nov 02 02:16:41.447986 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:41.447959    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:41.467655 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:41.466756    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 02:16:41.467655 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:41.466827    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 02:16:41.575245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:41.575176    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podStartSLOduration=2.5751591190000003 podStartE2EDuration="2.575159119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:16:41.553062852 +0000 UTC m=+3109.426703951" watchObservedRunningTime="2026-11-02 02:16:41.575159119 +0000 UTC m=+3109.448800172"
Nov 02 02:16:42.448268 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:42.448236    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 02:16:42.448706 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:42.448286    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 02:16:43.447773 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:43.447741    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 02:16:43.447955 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:43.447791    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 02:16:44.447942 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:44.447901    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 02:16:44.448358 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:44.447960    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 02:16:45.447941 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:45.447907    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 02:16:45.448147 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:45.447964    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 02:16:46.037074 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:16:46.037036    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:16:46.447532 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:46.447451    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 02:16:46.447532 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:46.447508    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 02:16:47.448155 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:47.448117    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 02 02:16:47.448631 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:47.448180    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 02 02:16:49.447995 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:49.447948    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:49.448479 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:49.448011    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:50.489827 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:50.489782    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:50.490259 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:50.489852    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:51.491251 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:51.491202    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:51.491809 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:51.491255    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:52.532792 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:52.532745    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:52.533281 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:52.532809    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:53.573799 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:53.573756    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:16:53.574205 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:53.573826    4287 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf" podUID="e26462c2-dfb3-4760-9f7b-ee4bd50c2e63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:16:53.610240 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:53.610210    4287 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:53.610519 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:53.610498    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:16:53.611090 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:16:53.611071    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-kmgsf"
Nov 02 02:17:16.037922 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:17:16.037880    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:17:46.037229 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:17:46.037188    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:18:16.038134 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:18:16.038096    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:18:46.037466 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:18:46.037430    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:18:56.565106 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:18:56.564709    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:18:56.565106 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:18:56.564793    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:18:57.176813 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:18:57.176764    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:18:57.177043 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:18:57.176841    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:19:16.037482 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:19:16.037440    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:19:46.037634 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:19:46.037573    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:19:52.748087 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:19:52.748059    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:19:52.778987 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:19:52.778962    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:20:16.037353 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:20:16.037313    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:20:46.038217 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:20:46.038178    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:21:16.037357 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:21:16.037320    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:21:27.185771 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:21:27.185715    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:21:27.186577 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:21:27.186541    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:21:46.037584 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:21:46.037547    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:22:16.039008 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:22:16.038972    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:22:46.037288 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:22:46.037250    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:22:46.133765 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.133733    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"]
Nov 02 02:22:46.134096 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.134081    4287 cpu_manager.go:414] "containerMap: removing container" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy"
Nov 02 02:22:46.134153 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.134098    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0589391c-05a1-4ccf-b816-9615d5a6d69f" containerName="istio-proxy"
Nov 02 02:22:46.137032 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.137018    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.189353 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.189321    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-fa3747-serving-cert\""
Nov 02 02:22:46.238170 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.238139    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmpbf\" (UniqueName: \"kubernetes.io/projected/8c3b7358-93cd-4421-b162-d0080570b187-kube-api-access-hmpbf\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.238347 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.238200    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3b7358-93cd-4421-b162-d0080570b187-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.238347 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.238267    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.238347 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.238306    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.241245 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.241218    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"]
Nov 02 02:22:46.339154 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.339051    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3b7358-93cd-4421-b162-d0080570b187-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.339154 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.339093    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.339427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.339345    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.339491 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.339466    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmpbf\" (UniqueName: \"kubernetes.io/projected/8c3b7358-93cd-4421-b162-d0080570b187-kube-api-access-hmpbf\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.339547 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:22:46.339483    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-serving-cert: secret "maas-api-e2e-worker-single-main-fa3747-serving-cert" not found
Nov 02 02:22:46.339595 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:22:46.339576    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls podName:8c3b7358-93cd-4421-b162-d0080570b187 nodeName:}" failed. No retries permitted until 2026-11-02 02:22:46.839551953 +0000 UTC m=+3474.713192982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls") pod "maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx" (UID: "8c3b7358-93cd-4421-b162-d0080570b187") : secret "maas-api-e2e-worker-single-main-fa3747-serving-cert" not found
Nov 02 02:22:46.339972 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.339949    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3b7358-93cd-4421-b162-d0080570b187-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.341753 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.341730    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.403886 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.403852    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmpbf\" (UniqueName: \"kubernetes.io/projected/8c3b7358-93cd-4421-b162-d0080570b187-kube-api-access-hmpbf\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.845042 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.845008    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:46.847427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:46.847404    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:47.049688 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:47.049656    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:47.422796 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:47.422740    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"]
Nov 02 02:22:47.425166 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:22:47.425134    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3b7358_93cd_4421_b162_d0080570b187.slice/crio-28aeddd5b81a63f639943b4708dbadb897502abeaa298dd1370b8cd33d191ded WatchSource:0}: Error finding container 28aeddd5b81a63f639943b4708dbadb897502abeaa298dd1370b8cd33d191ded: Status 404 returned error can't find the container with id 28aeddd5b81a63f639943b4708dbadb897502abeaa298dd1370b8cd33d191ded
Nov 02 02:22:47.426667 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:47.426645    4287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:22:48.321148 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:48.321111    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx" event={"ID":"8c3b7358-93cd-4421-b162-d0080570b187","Type":"ContainerStarted","Data":"28aeddd5b81a63f639943b4708dbadb897502abeaa298dd1370b8cd33d191ded"}
Nov 02 02:22:49.327767 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:49.327737    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx" event={"ID":"8c3b7358-93cd-4421-b162-d0080570b187","Type":"ContainerStarted","Data":"d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836"}
Nov 02 02:22:49.328257 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:49.327799    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:22:49.445597 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:49.445544    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx" podStartSLOduration=2.5417498419999998 podStartE2EDuration="3.445531246s" totalImagesPullingTime="903.781404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:22:49.444327026 +0000 UTC m=+3477.317968079" watchObservedRunningTime="2026-11-02 02:22:49.445531246 +0000 UTC m=+3477.319172298"
Nov 02 02:22:55.337919 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:22:55.337848    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:23:16.039142 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:23:16.039102    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:23:46.040294 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:23:46.040250    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:24:16.037760 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:24:16.037725    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:24:22.460555 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:22.460520    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"]
Nov 02 02:24:22.461107 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:22.460755    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx" podUID="8c3b7358-93cd-4421-b162-d0080570b187" containerName="maas-api" containerID="cri-o://d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836" gracePeriod=30
Nov 02 02:24:23.718684 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.718653    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:24:23.789836 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.789799    4287 generic.go:356] "Generic (PLEG): container finished" podID="8c3b7358-93cd-4421-b162-d0080570b187" containerID="d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836" exitCode=0
Nov 02 02:24:23.790006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.789843    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"
Nov 02 02:24:23.790006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.789886    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx" event={"ID":"8c3b7358-93cd-4421-b162-d0080570b187","Type":"ContainerDied","Data":"d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836"}
Nov 02 02:24:23.790006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.789959    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx" event={"ID":"8c3b7358-93cd-4421-b162-d0080570b187","Type":"ContainerDied","Data":"28aeddd5b81a63f639943b4708dbadb897502abeaa298dd1370b8cd33d191ded"}
Nov 02 02:24:23.790006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.789984    4287 scope.go:122] "RemoveContainer" containerID="d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836"
Nov 02 02:24:23.799190 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.799169    4287 scope.go:122] "RemoveContainer" containerID="d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836"
Nov 02 02:24:23.799528 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:24:23.799502    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836\": container with ID starting with d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836 not found: ID does not exist" containerID="d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836"
Nov 02 02:24:23.799571 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.799540    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836"} err="failed to get container status \"d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836\": rpc error: code = NotFound desc = could not find container \"d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836\": container with ID starting with d42d88fbf768abd92b6465bc534c6d0227cdafe66ea32a4feaa7d72aa90e9836 not found: ID does not exist"
Nov 02 02:24:23.859006 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.858972    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls\") pod \"8c3b7358-93cd-4421-b162-d0080570b187\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") "
Nov 02 02:24:23.859175 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.859049    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-metrics-certs\") pod \"8c3b7358-93cd-4421-b162-d0080570b187\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") "
Nov 02 02:24:23.859175 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.859104    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3b7358-93cd-4421-b162-d0080570b187-kube-api-access-hmpbf\" (UniqueName: \"kubernetes.io/projected/8c3b7358-93cd-4421-b162-d0080570b187-kube-api-access-hmpbf\") pod \"8c3b7358-93cd-4421-b162-d0080570b187\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") "
Nov 02 02:24:23.859175 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.859146    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3b7358-93cd-4421-b162-d0080570b187-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3b7358-93cd-4421-b162-d0080570b187-openshift-service-ca\") pod \"8c3b7358-93cd-4421-b162-d0080570b187\" (UID: \"8c3b7358-93cd-4421-b162-d0080570b187\") "
Nov 02 02:24:23.859706 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.859668    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3b7358-93cd-4421-b162-d0080570b187-openshift-service-ca" pod "8c3b7358-93cd-4421-b162-d0080570b187" (UID: "8c3b7358-93cd-4421-b162-d0080570b187"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:24:23.861505 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.861471    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3b7358-93cd-4421-b162-d0080570b187-kube-api-access-hmpbf" pod "8c3b7358-93cd-4421-b162-d0080570b187" (UID: "8c3b7358-93cd-4421-b162-d0080570b187"). InnerVolumeSpecName "kube-api-access-hmpbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:24:23.861734 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.861705    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls" pod "8c3b7358-93cd-4421-b162-d0080570b187" (UID: "8c3b7358-93cd-4421-b162-d0080570b187"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:24:23.861823 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.861780    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-metrics-certs" pod "8c3b7358-93cd-4421-b162-d0080570b187" (UID: "8c3b7358-93cd-4421-b162-d0080570b187"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:24:23.960081 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.960041    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:23.960081 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.960074    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c3b7358-93cd-4421-b162-d0080570b187-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:23.960081 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.960085    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmpbf\" (UniqueName: \"kubernetes.io/projected/8c3b7358-93cd-4421-b162-d0080570b187-kube-api-access-hmpbf\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:23.960333 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:23.960096    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3b7358-93cd-4421-b162-d0080570b187-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:24:24.210808 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:24.210770    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"]
Nov 02 02:24:24.214551 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:24.214527    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fa3747-7b675c4796-khjqx"]
Nov 02 02:24:24.236197 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:24.236165    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3b7358-93cd-4421-b162-d0080570b187" path="/var/lib/kubelet/pods/8c3b7358-93cd-4421-b162-d0080570b187/volumes"
Nov 02 02:24:46.039816 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:24:46.039777    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:24:52.804867 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:52.804839    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:24:52.822439 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:24:52.822416    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:25:16.038297 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:25:16.038254    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:25:46.038519 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:25:46.038480    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:26:16.038156 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:26:16.038116    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:26:46.038051 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:26:46.038012    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:27:16.037769 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:27:16.037729    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:27:27.919574 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.919534    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"]
Nov 02 02:27:27.923239 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.920794    4287 cpu_manager.go:414] "containerMap: removing container" podUID="8c3b7358-93cd-4421-b162-d0080570b187" containerName="maas-api"
Nov 02 02:27:27.923239 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.920822    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3b7358-93cd-4421-b162-d0080570b187" containerName="maas-api"
Nov 02 02:27:27.923239 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.921084    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3b7358-93cd-4421-b162-d0080570b187" containerName="maas-api"
Nov 02 02:27:27.925731 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.925700    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:27.993422 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.993392    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:27.993582 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.993427    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:27.993582 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.993509    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nzr\" (UniqueName: \"kubernetes.io/projected/2de84cdb-c892-4a0c-a8dc-788f72103650-kube-api-access-z8nzr\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:27.993697 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:27.993624    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de84cdb-c892-4a0c-a8dc-788f72103650-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.027889 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.027860    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3ffe96-serving-cert\""
Nov 02 02:27:28.030213 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.030190    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"]
Nov 02 02:27:28.094598 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.094562    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nzr\" (UniqueName: \"kubernetes.io/projected/2de84cdb-c892-4a0c-a8dc-788f72103650-kube-api-access-z8nzr\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.094802 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.094710    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de84cdb-c892-4a0c-a8dc-788f72103650-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.094802 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.094759    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.094802 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.094784    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.094973 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:27:28.094892    4287 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-serving-cert: secret "maas-api-e2e-worker-single-main-3ffe96-serving-cert" not found
Nov 02 02:27:28.095036 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:27:28.094978    4287 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls podName:2de84cdb-c892-4a0c-a8dc-788f72103650 nodeName:}" failed. No retries permitted until 2026-11-02 02:27:28.594957289 +0000 UTC m=+3756.468598334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls") pod "maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979" (UID: "2de84cdb-c892-4a0c-a8dc-788f72103650") : secret "maas-api-e2e-worker-single-main-3ffe96-serving-cert" not found
Nov 02 02:27:28.095450 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.095427    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de84cdb-c892-4a0c-a8dc-788f72103650-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.097472 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.097446    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.156225 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.156191    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nzr\" (UniqueName: \"kubernetes.io/projected/2de84cdb-c892-4a0c-a8dc-788f72103650-kube-api-access-z8nzr\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.599828 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.599782    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.602500 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.602470    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:28.841797 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:28.841757    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:29.215256 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:29.215229    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"]
Nov 02 02:27:29.217191 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:27:29.217155    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de84cdb_c892_4a0c_a8dc_788f72103650.slice/crio-556e235b0f167ccd70044ed32b1d9d151d89fbc613232fe686eb316aa5d5cbbf WatchSource:0}: Error finding container 556e235b0f167ccd70044ed32b1d9d151d89fbc613232fe686eb316aa5d5cbbf: Status 404 returned error can't find the container with id 556e235b0f167ccd70044ed32b1d9d151d89fbc613232fe686eb316aa5d5cbbf
Nov 02 02:27:29.696878 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:29.696843    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979" event={"ID":"2de84cdb-c892-4a0c-a8dc-788f72103650","Type":"ContainerStarted","Data":"556e235b0f167ccd70044ed32b1d9d151d89fbc613232fe686eb316aa5d5cbbf"}
Nov 02 02:27:30.704067 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:30.704034    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979" event={"ID":"2de84cdb-c892-4a0c-a8dc-788f72103650","Type":"ContainerStarted","Data":"f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302"}
Nov 02 02:27:30.704473 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:30.704159    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:30.823524 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:30.823461    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979" podStartSLOduration=2.901785796 podStartE2EDuration="3.823446945s" totalImagesPullingTime="921.661149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:27:30.821673018 +0000 UTC m=+3758.695314071" watchObservedRunningTime="2026-11-02 02:27:30.823446945 +0000 UTC m=+3758.697087997"
Nov 02 02:27:36.714107 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:27:36.714080    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:27:46.037892 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:27:46.037777    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:28:16.038307 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:28:16.038260    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:28:46.038589 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:28:46.038544    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:28:54.763209 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:54.763164    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"]
Nov 02 02:28:54.763831 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:54.763426    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979" podUID="2de84cdb-c892-4a0c-a8dc-788f72103650" containerName="maas-api" containerID="cri-o://f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302" gracePeriod=30
Nov 02 02:28:56.012699 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.012674    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:28:56.100040 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.099949    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls\") pod \"2de84cdb-c892-4a0c-a8dc-788f72103650\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") "
Nov 02 02:28:56.100228 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.100056    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de84cdb-c892-4a0c-a8dc-788f72103650-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de84cdb-c892-4a0c-a8dc-788f72103650-openshift-service-ca\") pod \"2de84cdb-c892-4a0c-a8dc-788f72103650\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") "
Nov 02 02:28:56.100228 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.100091    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de84cdb-c892-4a0c-a8dc-788f72103650-kube-api-access-z8nzr\" (UniqueName: \"kubernetes.io/projected/2de84cdb-c892-4a0c-a8dc-788f72103650-kube-api-access-z8nzr\") pod \"2de84cdb-c892-4a0c-a8dc-788f72103650\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") "
Nov 02 02:28:56.100228 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.100112    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-metrics-certs\") pod \"2de84cdb-c892-4a0c-a8dc-788f72103650\" (UID: \"2de84cdb-c892-4a0c-a8dc-788f72103650\") "
Nov 02 02:28:56.100440 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.100415    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de84cdb-c892-4a0c-a8dc-788f72103650-openshift-service-ca" pod "2de84cdb-c892-4a0c-a8dc-788f72103650" (UID: "2de84cdb-c892-4a0c-a8dc-788f72103650"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:28:56.102270 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.102248    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-metrics-certs" pod "2de84cdb-c892-4a0c-a8dc-788f72103650" (UID: "2de84cdb-c892-4a0c-a8dc-788f72103650"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:28:56.102367 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.102351    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls" pod "2de84cdb-c892-4a0c-a8dc-788f72103650" (UID: "2de84cdb-c892-4a0c-a8dc-788f72103650"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:28:56.102411 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.102377    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de84cdb-c892-4a0c-a8dc-788f72103650-kube-api-access-z8nzr" pod "2de84cdb-c892-4a0c-a8dc-788f72103650" (UID: "2de84cdb-c892-4a0c-a8dc-788f72103650"). InnerVolumeSpecName "kube-api-access-z8nzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:28:56.128774 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.128743    4287 generic.go:356] "Generic (PLEG): container finished" podID="2de84cdb-c892-4a0c-a8dc-788f72103650" containerID="f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302" exitCode=0
Nov 02 02:28:56.128916 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.128809    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"
Nov 02 02:28:56.128916 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.128791    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979" event={"ID":"2de84cdb-c892-4a0c-a8dc-788f72103650","Type":"ContainerDied","Data":"f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302"}
Nov 02 02:28:56.129013 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.128994    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979" event={"ID":"2de84cdb-c892-4a0c-a8dc-788f72103650","Type":"ContainerDied","Data":"556e235b0f167ccd70044ed32b1d9d151d89fbc613232fe686eb316aa5d5cbbf"}
Nov 02 02:28:56.129051 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.129038    4287 scope.go:122] "RemoveContainer" containerID="f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302"
Nov 02 02:28:56.141364 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.141334    4287 scope.go:122] "RemoveContainer" containerID="f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302"
Nov 02 02:28:56.141709 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:28:56.141690    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302\": container with ID starting with f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302 not found: ID does not exist" containerID="f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302"
Nov 02 02:28:56.141812 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.141719    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302"} err="failed to get container status \"f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302\": rpc error: code = NotFound desc = could not find container \"f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302\": container with ID starting with f25d9c18a8a2c8cd2fcfb23cf920b68b133cfa3a765704459fe2638f1148f302 not found: ID does not exist"
Nov 02 02:28:56.201423 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.201393    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.201423 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.201419    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de84cdb-c892-4a0c-a8dc-788f72103650-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.201423 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.201430    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8nzr\" (UniqueName: \"kubernetes.io/projected/2de84cdb-c892-4a0c-a8dc-788f72103650-kube-api-access-z8nzr\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.201671 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.201441    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2de84cdb-c892-4a0c-a8dc-788f72103650-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:28:56.297798 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.297765    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"]
Nov 02 02:28:56.304993 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:56.304965    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ffe96-94fff8f4f-28979"]
Nov 02 02:28:58.236833 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:28:58.236791    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de84cdb-c892-4a0c-a8dc-788f72103650" path="/var/lib/kubelet/pods/2de84cdb-c892-4a0c-a8dc-788f72103650/volumes"
Nov 02 02:29:16.042435 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:29:16.042399    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:29:46.037925 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:29:46.037890    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:29:52.845431 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:29:52.845402    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:29:52.862256 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:29:52.862238    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:30:16.040458 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:30:16.040416    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:30:26.563227 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:30:26.562726    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:30:26.563227 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:30:26.562797    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-p5tbl" podUID="e43fbecd-4574-4bcb-b700-7d9da3d08656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:30:27.182764 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:30:27.182725    4287 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 02:30:27.183119 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:30:27.183065    4287 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8x8rj" podUID="8f3e385c-906e-4f68-855a-f299826eb33b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 02:30:46.037568 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:30:46.037532    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:31:16.039481 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:31:16.039447    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:31:26.296691 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.296650    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-644c59f4c-64858"]
Nov 02 02:31:26.297075 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.297009    4287 cpu_manager.go:414] "containerMap: removing container" podUID="2de84cdb-c892-4a0c-a8dc-788f72103650" containerName="maas-api"
Nov 02 02:31:26.297075 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.297021    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de84cdb-c892-4a0c-a8dc-788f72103650" containerName="maas-api"
Nov 02 02:31:26.297147 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.297089    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de84cdb-c892-4a0c-a8dc-788f72103650" containerName="maas-api"
Nov 02 02:31:26.300226 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.300205    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.400545 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.400508    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-openshift-service-ca\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.400768 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.400660    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-maas-api-tls\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.400768 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.400707    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wtt6\" (UniqueName: \"kubernetes.io/projected/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-kube-api-access-2wtt6\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.400768 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.400737    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-metrics-certs\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.404068 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.404049    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-644c59f4c-64858"]
Nov 02 02:31:26.501796 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.501760    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-maas-api-tls\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.501984 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.501808    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wtt6\" (UniqueName: \"kubernetes.io/projected/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-kube-api-access-2wtt6\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.501984 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.501827    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-metrics-certs\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.501984 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.501880    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-openshift-service-ca\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.502498 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.502480    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-openshift-service-ca\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.504427 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.504403    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-metrics-certs\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.504517 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.504442    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-maas-api-tls\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.562758 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.562686    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wtt6\" (UniqueName: \"kubernetes.io/projected/b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7-kube-api-access-2wtt6\") pod \"maas-api-644c59f4c-64858\" (UID: \"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.613632 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.613573    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:26.989538 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.989515    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-644c59f4c-64858"]
Nov 02 02:31:26.991523 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:31:26.991496    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99d7ea2_8c2a_46cb_b748_3d5c6e392cc7.slice/crio-9e1b5b6c7fcc6ff365852f60c9c05eaec61cb0cd287a4fd50c56906c225dcbbb WatchSource:0}: Error finding container 9e1b5b6c7fcc6ff365852f60c9c05eaec61cb0cd287a4fd50c56906c225dcbbb: Status 404 returned error can't find the container with id 9e1b5b6c7fcc6ff365852f60c9c05eaec61cb0cd287a4fd50c56906c225dcbbb
Nov 02 02:31:26.992936 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:26.992915    4287 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 02:31:27.938676 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:27.938639    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858" event={"ID":"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7","Type":"ContainerStarted","Data":"9e1b5b6c7fcc6ff365852f60c9c05eaec61cb0cd287a4fd50c56906c225dcbbb"}
Nov 02 02:31:28.945191 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:28.945153    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858" event={"ID":"b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7","Type":"ContainerStarted","Data":"fd0dc587169aa883a3cc9e9e3e63da14a5947c5b655865138a2620f648fd814e"}
Nov 02 02:31:28.945584 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:28.945285    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:29.063894 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:29.063841    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858" podStartSLOduration=1.869053595 podStartE2EDuration="3.063824365s" totalImagesPullingTime="1.19477077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:29.062348053 +0000 UTC m=+3996.935989106" watchObservedRunningTime="2026-11-02 02:31:29.063824365 +0000 UTC m=+3996.937465417"
Nov 02 02:31:34.955760 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:34.955680    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-644c59f4c-64858"
Nov 02 02:31:44.175898 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:44.175862    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"]
Nov 02 02:31:44.176423 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:44.176114    4287 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62" podUID="eead6b71-0ae4-4224-a254-42d2b6008d0c" containerName="maas-api" containerID="cri-o://bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f" gracePeriod=30
Nov 02 02:31:45.422694 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.422669    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 02:31:45.480172 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.480092    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eead6b71-0ae4-4224-a254-42d2b6008d0c-kube-api-access-7wtfk\" (UniqueName: \"kubernetes.io/projected/eead6b71-0ae4-4224-a254-42d2b6008d0c-kube-api-access-7wtfk\") pod \"eead6b71-0ae4-4224-a254-42d2b6008d0c\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") "
Nov 02 02:31:45.480172 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.480145    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-maas-api-tls\") pod \"eead6b71-0ae4-4224-a254-42d2b6008d0c\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") "
Nov 02 02:31:45.480402 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.480201    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eead6b71-0ae4-4224-a254-42d2b6008d0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eead6b71-0ae4-4224-a254-42d2b6008d0c-openshift-service-ca\") pod \"eead6b71-0ae4-4224-a254-42d2b6008d0c\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") "
Nov 02 02:31:45.480402 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.480222    4287 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-metrics-certs\") pod \"eead6b71-0ae4-4224-a254-42d2b6008d0c\" (UID: \"eead6b71-0ae4-4224-a254-42d2b6008d0c\") "
Nov 02 02:31:45.480624 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.480584    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eead6b71-0ae4-4224-a254-42d2b6008d0c-openshift-service-ca" pod "eead6b71-0ae4-4224-a254-42d2b6008d0c" (UID: "eead6b71-0ae4-4224-a254-42d2b6008d0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 02:31:45.482458 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.482429    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-maas-api-tls" pod "eead6b71-0ae4-4224-a254-42d2b6008d0c" (UID: "eead6b71-0ae4-4224-a254-42d2b6008d0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:31:45.482571 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.482555    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eead6b71-0ae4-4224-a254-42d2b6008d0c-kube-api-access-7wtfk" pod "eead6b71-0ae4-4224-a254-42d2b6008d0c" (UID: "eead6b71-0ae4-4224-a254-42d2b6008d0c"). InnerVolumeSpecName "kube-api-access-7wtfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 02:31:45.482639 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.482565    4287 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-metrics-certs" pod "eead6b71-0ae4-4224-a254-42d2b6008d0c" (UID: "eead6b71-0ae4-4224-a254-42d2b6008d0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 02:31:45.580878 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.580844    4287 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wtfk\" (UniqueName: \"kubernetes.io/projected/eead6b71-0ae4-4224-a254-42d2b6008d0c-kube-api-access-7wtfk\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:45.580878 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.580879    4287 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-maas-api-tls\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:45.581065 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.580889    4287 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eead6b71-0ae4-4224-a254-42d2b6008d0c-openshift-service-ca\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:45.581065 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:45.580898    4287 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eead6b71-0ae4-4224-a254-42d2b6008d0c-metrics-certs\") on node \"ip-10-0-134-183.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 02:31:46.027846 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.027812    4287 generic.go:356] "Generic (PLEG): container finished" podID="eead6b71-0ae4-4224-a254-42d2b6008d0c" containerID="bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f" exitCode=0
Nov 02 02:31:46.028027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.027878    4287 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"
Nov 02 02:31:46.028027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.027894    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62" event={"ID":"eead6b71-0ae4-4224-a254-42d2b6008d0c","Type":"ContainerDied","Data":"bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f"}
Nov 02 02:31:46.028027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.027951    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5655ff5-xfn62" event={"ID":"eead6b71-0ae4-4224-a254-42d2b6008d0c","Type":"ContainerDied","Data":"97bb288be44acebea86d7cb858563e521c075fa3ecff177d18c4e7cfe1395794"}
Nov 02 02:31:46.028027 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.027974    4287 scope.go:122] "RemoveContainer" containerID="bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f"
Nov 02 02:31:46.037625 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.037581    4287 scope.go:122] "RemoveContainer" containerID="bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f"
Nov 02 02:31:46.037942 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:31:46.037912    4287 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f\": container with ID starting with bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f not found: ID does not exist" containerID="bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f"
Nov 02 02:31:46.038051 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.037950    4287 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f"} err="failed to get container status \"bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f\": rpc error: code = NotFound desc = could not find container \"bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f\": container with ID starting with bf23788d441d52cee3c3bcf8c166aab8864981fe2bbe19ee0c3d5819e9f9969f not found: ID does not exist"
Nov 02 02:31:46.042174 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:31:46.042151    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:31:46.190910 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.190873    4287 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"]
Nov 02 02:31:46.195276 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.195248    4287 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5655ff5-xfn62"]
Nov 02 02:31:46.242730 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:31:46.242680    4287 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eead6b71-0ae4-4224-a254-42d2b6008d0c" path="/var/lib/kubelet/pods/eead6b71-0ae4-4224-a254-42d2b6008d0c/volumes"
Nov 02 02:32:16.037668 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:32:16.037635    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:32:46.038008 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:32:46.037901    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:33:16.038530 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:33:16.038485    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:33:46.038250 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:33:46.038214    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:34:16.038534 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:34:16.038500    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:34:46.038953 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:34:46.038904    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:34:52.882308 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:34:52.882273    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:34:52.895087 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:34:52.895065    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjszn_0406fb90-83ff-40c9-a3aa-e834abf84e46/ovn-acl-logging/0.log"
Nov 02 02:35:16.038423 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:35:16.038386    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:35:46.038787 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:35:46.038746    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:36:16.037256 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:36:16.037218    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:36:16.841150 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:16.841123    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tc4gc_b0da1661-c27c-4a4b-9331-87ff64a33683/manager/0.log"
Nov 02 02:36:17.173189 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:17.173118    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lqgxq_62e0a342-0ab5-4571-86ca-d8da29907012/manager/0.log"
Nov 02 02:36:18.426214 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:18.426181    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ww2c9_e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e/proxy/0.log"
Nov 02 02:36:19.176946 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:19.176915    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-644c59f4c-64858_b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7/maas-api/0.log"
Nov 02 02:36:20.451910 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:20.451882    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2fwct_cc5970e6-3332-4454-b542-290d640ae6fc/postgres/0.log"
Nov 02 02:36:22.779705 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:22.779675    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wxbbm_30ca19e2-0eef-41b4-b291-aa3d46a5f9ac/manager/0.log"
Nov 02 02:36:23.003923 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:23.003894    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-44hpr_f10be300-f0c2-485d-bea2-c8cd85512179/kuadrant-console-plugin/0.log"
Nov 02 02:36:24.868142 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:24.868095    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-xnjzm_d5a24b8b-043b-4029-bff2-a91708e86173/istio-proxy/0.log"
Nov 02 02:36:25.473504 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:25.473475    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79f447648c-49sv8_9edb95bd-6855-44f2-b04c-716f90f13af7/kube-auth-proxy/0.log"
Nov 02 02:36:26.154767 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:26.154740    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8x8rj_8f3e385c-906e-4f68-855a-f299826eb33b/istio-proxy/0.log"
Nov 02 02:36:27.156729 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:27.156701    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-kmgsf_e26462c2-dfb3-4760-9f7b-ee4bd50c2e63/istio-proxy/0.log"
Nov 02 02:36:27.434558 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:27.434490    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-p5tbl_e43fbecd-4574-4bcb-b700-7d9da3d08656/istio-proxy/0.log"
Nov 02 02:36:42.802852 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:42.802822    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tc4gc_b0da1661-c27c-4a4b-9331-87ff64a33683/manager/0.log"
Nov 02 02:36:43.168208 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:43.168135    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lqgxq_62e0a342-0ab5-4571-86ca-d8da29907012/manager/0.log"
Nov 02 02:36:44.376894 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:44.376861    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ww2c9_e591d4a5-c7fd-4ed6-bd4f-e0498a1ffd2e/proxy/0.log"
Nov 02 02:36:44.993790 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:44.993759    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-644c59f4c-64858_b99d7ea2-8c2a-46cb-b748-3d5c6e392cc7/maas-api/0.log"
Nov 02 02:36:46.038496 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:36:46.038455    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:36:46.283740 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:46.283708    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2fwct_cc5970e6-3332-4454-b542-290d640ae6fc/postgres/0.log"
Nov 02 02:36:48.267337 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:48.267308    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wxbbm_30ca19e2-0eef-41b4-b291-aa3d46a5f9ac/manager/0.log"
Nov 02 02:36:48.498552 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:48.498520    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-44hpr_f10be300-f0c2-485d-bea2-c8cd85512179/kuadrant-console-plugin/0.log"
Nov 02 02:36:50.023921 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:50.023883    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-xnjzm_d5a24b8b-043b-4029-bff2-a91708e86173/istio-proxy/0.log"
Nov 02 02:36:50.569948 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:50.569920    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79f447648c-49sv8_9edb95bd-6855-44f2-b04c-716f90f13af7/kube-auth-proxy/0.log"
Nov 02 02:36:51.232537 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:51.232501    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8x8rj_8f3e385c-906e-4f68-855a-f299826eb33b/istio-proxy/0.log"
Nov 02 02:36:52.237830 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:52.237800    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-kmgsf_e26462c2-dfb3-4760-9f7b-ee4bd50c2e63/istio-proxy/0.log"
Nov 02 02:36:52.501691 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:36:52.501602    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-p5tbl_e43fbecd-4574-4bcb-b700-7d9da3d08656/istio-proxy/0.log"
Nov 02 02:37:16.038467 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:37:16.038427    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:37:46.038547 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:37:46.038438    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:38:14.863234 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:14.863196    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hcbh5_1955308f-bad5-47ef-951a-95b7064ed5b2/global-pull-secret-syncer/0.log"
Nov 02 02:38:16.038598 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:38:16.038555    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:38:17.732278 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:17.732245    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vvj9k_3f612b3f-b624-45ae-b05d-2b4a164fd4af/konnectivity-agent/0.log"
Nov 02 02:38:18.549256 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:18.549216    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-183.us-west-1.compute.internal_8c8e4877a18ae50f2a48041b0698e786/haproxy/0.log"
Nov 02 02:38:24.450374 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.450335    4287 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"]
Nov 02 02:38:24.450776 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.450710    4287 cpu_manager.go:414] "containerMap: removing container" podUID="eead6b71-0ae4-4224-a254-42d2b6008d0c" containerName="maas-api"
Nov 02 02:38:24.450776 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.450722    4287 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eead6b71-0ae4-4224-a254-42d2b6008d0c" containerName="maas-api"
Nov 02 02:38:24.450854 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.450802    4287 memory_manager.go:365] "RemoveStaleState removing state" podUID="eead6b71-0ae4-4224-a254-42d2b6008d0c" containerName="maas-api"
Nov 02 02:38:24.454130 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.454111    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.483140 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.483117    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-wxbbm_30ca19e2-0eef-41b4-b291-aa3d46a5f9ac/manager/0.log"
Nov 02 02:38:24.506111 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.506079    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zt4ds\"/\"kube-root-ca.crt\""
Nov 02 02:38:24.506111 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.506105    4287 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zt4ds\"/\"openshift-service-ca.crt\""
Nov 02 02:38:24.533520 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.533492    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-proc\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.533683 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.533536    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-sys\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.533683 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.533553    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7m4\" (UniqueName: \"kubernetes.io/projected/4e74d018-a54c-4f11-b9de-d85d1deacbd0-kube-api-access-hq7m4\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.533760 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.533703    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-lib-modules\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.533760 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.533731    4287 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-podres\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.555863 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.555834    4287 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zt4ds\"/\"default-dockercfg-kqrg4\""
Nov 02 02:38:24.557932 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.557910    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"]
Nov 02 02:38:24.634751 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.634712    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-proc\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.634942 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.634767    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-sys\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.634942 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.634836    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-sys\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.634942 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.634838    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-proc\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.634942 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.634878    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7m4\" (UniqueName: \"kubernetes.io/projected/4e74d018-a54c-4f11-b9de-d85d1deacbd0-kube-api-access-hq7m4\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.634942 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.634936    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-lib-modules\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.635182 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.634956    4287 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-podres\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.635182 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.635052    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-podres\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.635182 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.635061    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e74d018-a54c-4f11-b9de-d85d1deacbd0-lib-modules\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.695514 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.695477    4287 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7m4\" (UniqueName: \"kubernetes.io/projected/4e74d018-a54c-4f11-b9de-d85d1deacbd0-kube-api-access-hq7m4\") pod \"perf-node-gather-daemonset-wx4q9\" (UID: \"4e74d018-a54c-4f11-b9de-d85d1deacbd0\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.766738 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.766705    4287 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:24.912456 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:24.912428    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-44hpr_f10be300-f0c2-485d-bea2-c8cd85512179/kuadrant-console-plugin/0.log"
Nov 02 02:38:25.139189 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:25.139157    4287 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"]
Nov 02 02:38:25.141458 ip-10-0-134-183 kubenswrapper[4287]: W1102 02:38:25.141436    4287 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e74d018_a54c_4f11_b9de_d85d1deacbd0.slice/crio-eb2dfe9ab71a2b1fbbc2ba1d6e6aa60a5deecbff89cfd20ff97ec130a294cb92 WatchSource:0}: Error finding container eb2dfe9ab71a2b1fbbc2ba1d6e6aa60a5deecbff89cfd20ff97ec130a294cb92: Status 404 returned error can't find the container with id eb2dfe9ab71a2b1fbbc2ba1d6e6aa60a5deecbff89cfd20ff97ec130a294cb92
Nov 02 02:38:25.995885 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:25.995847    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9" event={"ID":"4e74d018-a54c-4f11-b9de-d85d1deacbd0","Type":"ContainerStarted","Data":"31ba7b918739a40a41b6c47f38059e187b23ca5bdde121c345d5aa09addff791"}
Nov 02 02:38:25.996310 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:25.995901    4287 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9" event={"ID":"4e74d018-a54c-4f11-b9de-d85d1deacbd0","Type":"ContainerStarted","Data":"eb2dfe9ab71a2b1fbbc2ba1d6e6aa60a5deecbff89cfd20ff97ec130a294cb92"}
Nov 02 02:38:25.996310 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:25.995959    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:26.111633 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:26.111545    4287 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9" podStartSLOduration=2.111532678 podStartE2EDuration="2.111532678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:38:26.111030296 +0000 UTC m=+4413.984671348" watchObservedRunningTime="2026-11-02 02:38:26.111532678 +0000 UTC m=+4413.985173730"
Nov 02 02:38:32.011194 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:32.011165    4287 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-wx4q9"
Nov 02 02:38:33.868488 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:33.868444    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-p9wf7_8e58fc7d-197a-4856-8f1c-e0787c11ab2c/monitoring-plugin/0.log"
Nov 02 02:38:35.544055 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:35.544024    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xldxx_2f93bc86-436b-4eec-a84c-b391416a9844/node-exporter/0.log"
Nov 02 02:38:35.971453 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:35.971345    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xldxx_2f93bc86-436b-4eec-a84c-b391416a9844/kube-rbac-proxy/0.log"
Nov 02 02:38:36.385283 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:36.385258    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xldxx_2f93bc86-436b-4eec-a84c-b391416a9844/init-textfile/0.log"
Nov 02 02:38:39.331232 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:39.331194    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6aceefcc-71d6-40e5-bac0-bfed1932116a/prometheus/0.log"
Nov 02 02:38:39.746401 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:39.746327    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6aceefcc-71d6-40e5-bac0-bfed1932116a/config-reloader/0.log"
Nov 02 02:38:40.160325 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:40.160276    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6aceefcc-71d6-40e5-bac0-bfed1932116a/thanos-sidecar/0.log"
Nov 02 02:38:40.575726 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:40.575697    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6aceefcc-71d6-40e5-bac0-bfed1932116a/kube-rbac-proxy-web/0.log"
Nov 02 02:38:40.991465 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:40.991397    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6aceefcc-71d6-40e5-bac0-bfed1932116a/kube-rbac-proxy/0.log"
Nov 02 02:38:41.413689 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:41.413662    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6aceefcc-71d6-40e5-bac0-bfed1932116a/kube-rbac-proxy-thanos/0.log"
Nov 02 02:38:41.828068 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:41.828042    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6aceefcc-71d6-40e5-bac0-bfed1932116a/init-config-reloader/0.log"
Nov 02 02:38:42.253395 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:42.253301    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vmxbx_f92ba5d0-6d38-44d5-a41f-8d14274ca4f2/prometheus-operator/0.log"
Nov 02 02:38:42.665453 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:42.665422    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-vmxbx_f92ba5d0-6d38-44d5-a41f-8d14274ca4f2/kube-rbac-proxy/0.log"
Nov 02 02:38:46.038495 ip-10-0-134-183 kubenswrapper[4287]: E1102 02:38:46.038456    4287 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-29gbz" containerName="extractor"
Nov 02 02:38:47.472396 ip-10-0-134-183 kubenswrapper[4287]: I1102 02:38:47.472362    4287 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-xnjzm_d5a24b8b-043b-4029-bff2-a91708e86173/istio-proxy/0.log"